Update README.md with contribution guidelines.
Browse files
README.md
CHANGED
@@ -37,31 +37,50 @@ OFLL provides a specialized evaluation framework tailored specifically to the fi
|
|
37 |
The Open Financial LLM Leaderboard aims to set a new standard in evaluating the capabilities of language models in the financial domain, offering a specialized, real-world-focused benchmarking solution.
|
38 |
|
39 |
|
40 |
-
#
|
41 |
-
|
42 |
-
|
43 |
-
|
44 |
-
|
45 |
-
|
46 |
-
|
47 |
-
|
48 |
-
|
49 |
-
|
50 |
-
|
51 |
-
|
52 |
-
|
53 |
-
|
54 |
-
|
55 |
-
|
56 |
-
|
57 |
-
|
58 |
-
|
59 |
-
|
60 |
-
|
61 |
-
|
62 |
-
|
63 |
-
|
64 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
65 |
If you encounter problem on the space, don't hesitate to restart it to remove the create eval-queue, eval-queue-bk, eval-results and eval-results-bk created folder.
|
66 |
|
67 |
# Code logic for more complex edits
|
|
|
37 |
The Open Financial LLM Leaderboard aims to set a new standard in evaluating the capabilities of language models in the financial domain, offering a specialized, real-world-focused benchmarking solution.
|
38 |
|
39 |
|
40 |
+
# Contribute to OFLL
|
41 |
+
|
42 |
+
To make the leaderboard more accessible for external contributors, we offer clear guidelines for adding tasks, updating result files, and other maintenance activities.
|
43 |
+
|
44 |
+
1. **Primary Files**:
|
45 |
+
- `src/env.py`: Modify variables like repository paths for customization.
|
46 |
+
- `src/about.py`: Update task configurations here to add new datasets.
|
47 |
+
|
48 |
+
2. **Adding New Tasks**:
|
49 |
+
- Navigate to `src/about.py` and specify new tasks in the `Tasks` enum section.
|
50 |
+
- Each task requires details such as `benchmark`, `metric`, `col_name`, and `category`. For example:
|
51 |
+
```python
|
52 |
+
taskX = Task("DatasetName", "MetricType", "ColumnName", category="Category")
|
53 |
+
```
|
54 |
+
|
55 |
+
3. **Updating Results Files**:
|
56 |
+
- Results files should be in JSON format and structured as follows:
|
57 |
+
```json
|
58 |
+
{
|
59 |
+
"config": {
|
60 |
+
"model_dtype": "torch.float16",
|
61 |
+
"model_name": "path of the model on the hub: org/model",
|
62 |
+
"model_sha": "revision on the hub"
|
63 |
+
},
|
64 |
+
"results": {
|
65 |
+
"task_name": {
|
66 |
+
"metric_name": score
|
67 |
+
},
|
68 |
+
"task_name2": {
|
69 |
+
"metric_name": score
|
70 |
+
}
|
71 |
+
}
|
72 |
+
}
|
73 |
+
```
|
74 |
+
|
75 |
+
4. **Updating Leaderboard Data**:
|
76 |
+
- When a new task is added, ensure that the results JSON files reflect this update. This process will be automated in future releases.
|
77 |
+
- Access the current results at [Hugging Face Datasets](https://huggingface.co/datasets/TheFinAI/results/tree/main/demo-leaderboard).
|
78 |
+
|
79 |
+
5. **Useful Links**:
|
80 |
+
- [Hugging Face Leaderboard Documentation](https://huggingface.co/docs/leaderboards/en/leaderboards/building_page)
|
81 |
+
- [OFLL Demo on Hugging Face](https://huggingface.co/spaces/finosfoundation/Open-Financial-LLM-Leaderboard)
|
82 |
+
|
83 |
+
|
84 |
If you encounter problem on the space, don't hesitate to restart it to remove the create eval-queue, eval-queue-bk, eval-results and eval-results-bk created folder.
|
85 |
|
86 |
# Code logic for more complex edits
|