Add eval for gold-standard benchmarks

README.md

@@ -616,7 +616,243 @@ This instruction-tuned variant has been trained with a mixture of 276k English,
 
 ## Evaluation
 
-Details coming soon.
+### Gold-standard benchmarks
+
+Evaluation is done using the Language Model Evaluation Harness (Gao et al., 2024). We evaluate on a set of tasks taken from [SpanishBench](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/spanish_bench), [CatalanBench](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/catalan_bench), [BasqueBench](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/basque_bench) and [GalicianBench](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/galician_bench). These benchmarks include both new and existing tasks and datasets. Given that this is an instructed model, we add LM Evaluation Harness's native feature of `chat-template` to the setup. In the tables below, we include the results in a selection of evaluation datasets that represent model's performance across a variety of tasks within these benchmarks. 
+
+We only use tasks that are either human generated, human translated, or with a strong human-in-the-loop (i.e., machine translation followed by professional revision or machine generation followed by human revision and annotation). This is the reason behind the variety in number of tasks reported across languages. As more tasks that fulfill these requirements are published, we will update the presented results. We also intend to expand the evaluation to other languages, as long as the datasets meet our quality standards.
+
+During the implementation of the evaluation we observed a series of issues worth considering when replicating and interpreting the results presented. These issues include ≈1.5% variances in performance in some tasks depending on the version of the `transformers` library used, and depending on the use (or lack of use) of tensor parallelism when loading a model. When implementing existing tasks, we carry out a comprehensive quality evaluation of the dataset, the Harness task itself, and what kind of input models see during evaluation. Our implementation (see links above) addresses multiple existing problems such as errors in datasets and prompts, and lack of pre-processing. All this means that results will vary if using other Harness implementations, and may slightly vary depending on the replication setup.
+
+It should be noted that these results are subject to all the drawbacks of every current gold-standard evaluation, and that the figures do not fully represent the models capabilities and potential. We thus advise caution when reading and interpreting the results.
+
+A full list of results compared to other baselines, a discussion of the model's performance across tasks and its implications, and details regarding problem-solving with task implementation will soon be available in the technical report.
+
+All results reported below are on a 0-shot setting.
+
+#### Spanish
+
+<table><thead>
+  <tr>
+    <th>Category</th>
+    <th>Task</th>
+    <th>Metric</th>
+    <th>Result</th>
+  </tr></thead>
+<tbody>
+  <tr>
+    <td>Commonsense Reasoning</td>
+    <td>xstorycloze_es</td>
+    <td>acc</td>
+    <td>62.34</td>
+  </tr>
+  <tr>
+    <td rowspan="2">NLI</td>
+    <td>wnli_es</td>
+    <td>acc</td>
+    <td>47.89</td>
+  </tr>
+  <tr>
+    <td>xnli_es</td>
+    <td>acc</td>
+    <td>47.03</td>
+  </tr>
+  <tr>
+    <td>Paraphrasing</td>
+    <td>paws_es</td>
+    <td>acc</td>
+    <td>55.5</td>
+  </tr>
+  <tr>
+    <td>QA</td>
+    <td>xquad_es</td>
+    <td>acc</td>
+    <td>42.21</td>
+  </tr>
+  <tr>
+    <td>Translation</td>
+    <td>flores_es</td>
+    <td>bleu</td>
+    <td>20.27</td>
+  </tr>
+</tbody>
+</table>
+
+#### Catalan
+
+<table><thead>
+  <tr>
+    <th>Category</th>
+    <th>Task</th>
+    <th>Metric</th>
+    <th>Result</th>
+  </tr></thead>
+<tbody>
+  <tr>
+    <td rowspan="2">Commonsense Reasoning</td>
+    <td>copa_ca</td>
+    <td>acc</td>
+    <td>70.4</td>
+  </tr>
+  <tr>
+    <td>xstorycloze_ca</td>
+    <td>acc</td>
+    <td>63.07</td>
+  </tr>
+  <tr>
+    <td rowspan="2">NLI</td>
+    <td>wnli_ca</td>
+    <td>acc</td>
+    <td>52.11</td>
+  </tr>
+  <tr>
+    <td>xnli_ca</td>
+    <td>acc</td>
+    <td>51.69</td>
+  </tr>
+  <tr>
+    <td rowspan="2">Paraphrasing</td>
+    <td>parafraseja</td>
+    <td>acc</td>
+    <td>61.88</td>
+  </tr>
+  <tr>
+    <td>paws_ca</td>
+    <td>acc</td>
+    <td>57.7</td>
+  </tr>
+  <tr>
+    <td rowspan="5">QA</td>
+    <td>arc_ca_easy</td>
+    <td>acc</td>
+    <td>51.94</td>
+  </tr>
+  <tr>
+    <td>arc_ca_challenge</td>
+    <td>acc</td>
+    <td>29.52</td>
+  </tr>
+  <tr>
+    <td>openbookqa_ca</td>
+    <td>acc</td>
+    <td>26.4</td>
+  </tr>
+  <tr>
+    <td>piqa_ca</td>
+    <td>acc</td>
+    <td>62.89</td>
+  </tr>
+  <tr>
+    <td>siqa_ca</td>
+    <td>acc</td>
+    <td>42.63</td>
+  </tr>
+  <tr>
+    <td>Translation</td>
+    <td>flores_ca</td>
+    <td>bleu</td>
+    <td>24.48</td>
+  </tr>
+</tbody></table>
+
+#### Basque
+
+<table><thead>
+  <tr>
+    <th>Category</th>
+    <th>Task</th>
+    <th>Metric</th>
+    <th>Result</th>
+  </tr></thead>
+<tbody>
+  <tr>
+    <td rowspan="2">Commonsense Reasoning</td>
+    <td>xcopa_eu</td>
+    <td>acc</td>
+    <td>53.6</td>
+  </tr>
+  <tr>
+    <td>xstorycloze_eu</td>
+    <td>acc</td>
+    <td>56.39</td>
+  </tr>
+  <tr>
+    <td rowspan="2">NLI</td>
+    <td>wnli_eu</td>
+    <td>acc</td>
+    <td>45.07</td>
+  </tr>
+  <tr>
+    <td>xnli_eu</td>
+    <td>acc</td>
+    <td>39.44</td>
+  </tr>
+  <tr>
+    <td rowspan="3">QA</td>
+    <td>eus_exams</td>
+    <td>acc</td>
+    <td>25.35</td>
+  </tr>
+  <tr>
+    <td>eus_proficiency</td>
+    <td>acc</td>
+    <td>26.37</td>
+  </tr>
+  <tr>
+    <td>eus_trivia</td>
+    <td>acc</td>
+    <td>26.24</td>
+  </tr>
+  <tr>
+    <td>Reading Comprehension</td>
+    <td>eus_reading</td>
+    <td>acc</td>
+    <td>24.72</td>
+  </tr>
+  <tr>
+    <td>Translation</td>
+    <td>flores_eu</td>
+    <td>bleu</td>
+    <td>9.67</td>
+  </tr>
+</tbody></table>
+
+#### Galician
+
+<table><thead>
+  <tr>
+    <th>Category</th>
+    <th>Task</th>
+    <th>Metric</th>
+    <th>Result</th>
+  </tr></thead>
+<tbody>
+  <tr>
+    <td rowspan="2">Paraphrasing</td>
+    <td>parafrases_gl</td>
+    <td>acc</td>
+    <td>50.00</td>
+  </tr>
+  <tr>
+    <td>paws_gl</td>
+    <td>acc</td>
+    <td>52.20</td>
+  </tr>
+  <tr>
+    <td>QA</td>
+    <td>openbookqa_gl</td>
+    <td>acc</td>
+    <td>33.2</td>
+  </tr>
+  <tr>
+    <td>Translation</td>
+    <td>flores_gl</td>
+    <td>bleu</td>
+    <td>22.39</td>
+  </tr>
+</tbody>
+</table>
+
 
 ---