File size: 20,018 Bytes
e5ee086
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3c69c38
 
 
 
e5ee086
 
 
f728b4f
0e13466
f728b4f
0e13466
f728b4f
3c69c38
e5ee086
 
 
 
 
 
0e13466
 
 
 
e5ee086
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
e6aadde
e5ee086
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3c69c38
 
 
 
 
 
 
 
 
e5ee086
f728b4f
 
4f4b508
f7aa050
4f4b508
 
9b7ca38
4f4b508
9b7ca38
 
 
 
 
f728b4f
 
 
f7aa050
4f4b508
 
 
 
 
 
 
 
 
 
 
 
 
f728b4f
7853dd1
 
 
 
09c4409
 
 
 
 
 
 
 
 
 
 
 
0f6dbd0
 
 
5ec15e0
 
 
 
5155380
 
 
565a777
4c295e7
 
565a777
 
 
7853dd1
 
a4ad90d
 
7853dd1
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
from dataclasses import dataclass
from enum import Enum


@dataclass
class Task:
    benchmark: str
    metric: str
    col_name: str


# Select your tasks here
# ---------------------------------------------------
class Tasks(Enum):
    # task_key in the json file, metric_key in the json file, name to display in the leaderboard
    task0 = Task("main", "acc", "Overall")


NUM_FEWSHOT = 0  # Change with your few shot
# ---------------------------------------------------


# Your leaderboard name
TITLE = """<h1 align="center" id="space-title">πŸ‘©β€πŸ«Invalsi Leaderboard
</h1>"""

"""
<img src="https://huggingface.co/spaces/Crisp-Unimib/INVALSIbenchmark/resolve/main/src/logo-crisp-eng-retina.png" height="800" width="150" style="vertical-align: middle;">
"""

# What does your leaderboard evaluate?
INTRODUCTION_TEXT = """
Welcome into <a href="https://crispresearch.it/"><b>CRISP Bicocca</b></a> Invalsi Leaderboard page! 

We adapted the INVALSI benchmark for automated LLM evaluation, which involves rigorous adaptation of the test format to suit automated processing while retaining the essence of the original tests. In this leaderboard, we provide a detailed assessment of current LLMs, offering a crucial reference point for the academic community.

Researchers are invited to submit their models for ongoing evaluation, ensuring the benchmark remains a current and valuable resource.

For more information on the benchmark, please refer to our arXiv paper <a href="https://arxiv.org/abs/2406.17535"><b>here</b></a> and read the "About" section.
"""

# Which evaluations are you running? how can people reproduce what you have?
LLM_BENCHMARKS_TEXT = f"""
## INVALSI Benchmark 

The INVALSI (National Institute for the Evaluation of the Education and Training System) test has been essential to Italy's educational assessment framework since the 2005-2006 academic year. It aims to evaluate students' competencies across various subjects, including Italian language and mathematics, at multiple stages of the education system in a standardised manner. The test's primary objective is to assess students' linguistic proficiency, a crucial skill that schools are tasked with developing as per national educational guidelines.

Since the test covers a wide range of linguistic and comprehension skills, using it to evaluate LLMs can provide a detailed view of a model's proficiency in handling real-world, nuanced language tasks designed for human learners. The test's structured and standardised nature makes it an excellent benchmark for comparing different LLMs with questions culturally and contextually relevant to Italian speakers. However, our findings are relevant across all languages since they assess various general capabilities, such as word formation and text comprehension abilities. Also, since it is designed for different educational stages, it provides varied levels of complexity and challenge. This aspect can gauge an LLM's capability at different difficulty levels, reflecting its potential scalability and adaptability across simpler to more complex linguistic tasks.

### Data Collection
We have collected from public sources 58 unique tests, divided into 141 unique units, with 2114 questions and 2808 unique items. Some questions are subdivided into multiple items, each requiring a specific answer. 
The data has been gathered from the <a href="https://www.gestinv.it/Index.aspx"><b>Gestinv</b></a> database.
Questions' formatting is sometimes not adequately structured for LLM evaluation; for instance, it is sometimes impossible to automatically transcribe the questions into structured fields, necessitating further inspection of images and PDFs. For this reason, we also collected corresponding PDF files and images. Manual inspection was required to ensure accuracy. In cases where questions involved graphical elements, we modified them into an appropriate multiple-choice format. For example, if the task required a student to find and underscore a word, we reformulated the question to allow selection from multiple choices. Similarly, if the task involved drawing a line between two groups of conceptsβ€”a common task for younger studentsβ€”we rephrased it to involve choosing the correct association from given options. Generally, we aimed to adapt the questions to a format that allows the model to select the correct answer from a pool of choices if it aligns with the original type of question.


### Dataset Characteristics

<ul>
 <li> We have selected <b>10 tests</b> comprising <b>31 unique units</b>, <b>409 questions</b>, and <b>625 items</b> from the above data. A test comprises <b>two or more different units</b>; each question can have more than one item to answer. The sample of tests has been chosen by manual inspection, aiming to include different grades and years and avoiding those with questions that require inspecting an image or contained questions that would be hard to reformulate for language model comprehension. <br> <br>
<table style="font-size: 18px; border-collapse: collapse; width: 60%; margin: auto; border: 1px solid black;">
    <caption style="caption-side: top; text-align: center; font-weight: bold;">
        Distribution of tests, questions and items by educational grade.
    </caption>
    <thead>
        <tr style="background-color: #f2f2f2;">
            <th style="border: 1px solid black; padding: 8px; text-align: left;">School Grade</th>
            <th style="border: 1px solid black; padding: 8px; text-align: left;"># Tests</th>
            <th style="border: 1px solid black; padding: 8px; text-align: left;"># Questions</th>
            <th style="border: 1px solid black; padding: 8px; text-align: left;"># Items</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">2nd Grade (Primary School)</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">2</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">34</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">72</td>
        </tr>
        <tr>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">5th Grade (Primary School)</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">2</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">75</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">117</td>
        </tr>
        <tr>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">6th Grade (Middle School)</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">1</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">87</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">118</td>
        </tr>
        <tr>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">8th Grade (Middle School)</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">2</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">86</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">88</td>
        </tr>
        <tr>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">10th Grade (High School)</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">2</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">77</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">139</td>
        </tr>
        <tr>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">13th Grade (High School)</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">1</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">50</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">91</td>
        </tr>
    </tbody>
</table> 
</li>
<br> <br>
<li> Each question in the benchmark is labelled with the specific <b>lexical macro area</b> it aims to assess <br> <br>

<table style="font-size: 18px; border-collapse: collapse; width: 80%; margin: auto; border: 1px solid black;">
    <caption style="caption-side: top; text-align: center; font-weight: bold;">
        Distribution of questions by section and macro area.
    </caption>
    <thead>
        <tr style="background-color: #f2f2f2;">
            <th style="border: 1px solid black; padding: 8px; text-align: left;">Section</th>
            <th style="border: 1px solid black; padding: 8px; text-align: left; width: 3.5cm;">Macro Area</th>
            <th style="border: 1px solid black; padding: 8px; text-align: right;"># Questions</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td rowspan="3" style="border: 1px solid black; padding: 8px; text-align: left; width: 1.3cm;">Text comprehension</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">Reconstruct the meaning of the text, locally or globally</td>
            <td style="border: 1px solid black; padding: 8px; text-align: right;">179 (43.8%)</td>
        </tr>
        <tr>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">Locate and identify information within the text</td>
            <td style="border: 1px solid black; padding: 8px; text-align: right;">108 (26.4%)</td>
        </tr>
        <tr>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">Reflect on the content or form of the text, <br> locally or globally, and evaluate them</td>
            <td style="border: 1px solid black; padding: 8px; text-align: right;">33 (8.1%)</td>
        </tr>
        <tr>
            <td rowspan="6" style="border: 1px solid black; padding: 8px; text-align: left; width: 1.3cm;">Reflection on the language</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">Lexicon and semantics</td>
            <td style="border: 1px solid black; padding: 8px; text-align: right;">29 (7.1%)</td>
        </tr>
        <tr>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">Morphology</td>
            <td style="border: 1px solid black; padding: 8px; text-align: right;">24 (5.9%)</td>
        </tr>
        <tr>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">Syntax</td>
            <td style="border: 1px solid black; padding: 8px; text-align: right;">19 (4.6%)</td>
        </tr>
        <tr>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">Word formation</td>
            <td style="border: 1px solid black; padding: 8px; text-align: right;">7 (1.7%)</td>
        </tr>
        <tr>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">Textuality and pragmatics</td>
            <td style="border: 1px solid black; padding: 8px; text-align: right;">6 (1.5%)</td>
        </tr>
        <tr>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">Spelling</td>
            <td style="border: 1px solid black; padding: 8px; text-align: right;">4 (1.0%)</td>
        </tr>
    </tbody>
</table>

</li>
<br> <br>

<li> The questions come in five distinct formats <br> <br> 
<table style="width: 90%; margin: auto; border-collapse: collapse; font-size: 18px; border: 1px solid black;">
    <caption style="caption-side: top; text-align: center; font-weight: bold;">
        Combined Legend of Question Types and Distribution by Format
    </caption>
    <thead>
        <tr style="background-color: #f2f2f2;">
            <th style="border: 1px solid black; padding: 10px; text-align: center;">Question Type</th>
            <th style="border: 1px solid black; padding: 10px; text-align: center;">Description</th>
            <th style="border: 1px solid black; padding: 10px; text-align: center;"># Questions</th>
            <th style="border: 1px solid black; padding: 10px; text-align: center;"># Items</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td style="border: 1px solid black; padding: 10px; text-align: left;"><i>Multiple Choice (MC)</i></td>
            <td style="border: 1px solid black; padding: 10px; text-align: left;">Composed of a question with several answer options, among which only one is correct. The most common question format.</td>
            <td style="border: 1px solid black; padding: 10px; text-align: left;">334 (81.7%)</td>
            <td style="border: 1px solid black; padding: 10px; text-align: left;">340</td>
        </tr>
        <tr>
            <td style="border: 1px solid black; padding: 10px; text-align: left;"><i>Multiple Complex Choice (MCC)</i></td>
            <td style="border: 1px solid black; padding: 10px; text-align: left;">Composed of input questions and multiple items to answer. Requires all items to be correctly answered.</td>
            <td style="border: 1px solid black; padding: 10px; text-align: left;">38 (9.3%)</td>
            <td style="border: 1px solid black; padding: 10px; text-align: left;">228</td>
        </tr>
        <tr>
            <td style="border: 1px solid black; padding: 10px; text-align: left;"><i>Unique Response (RU)</i></td>
            <td style="border: 1px solid black; padding: 10px; text-align: left;">Open-ended questions with only one correct answer, sometimes allowing limited variants.</td>
            <td style="border: 1px solid black; padding: 10px; text-align: left;">33 (8.1%)</td>
            <td style="border: 1px solid black; padding: 10px; text-align: left;">50</td>
        </tr>
        <tr>
            <td style="border: 1px solid black; padding: 10px; text-align: left;"><i>Short Response (RB)</i></td>
            <td style="border: 1px solid black; padding: 10px; text-align: left;">Specific sub-type of RU questions requiring a single identifying word as the answer.</td>
            <td style="border: 1px solid black; padding: 10px; text-align: left;">1 (0.2%)</td>
            <td style="border: 1px solid black; padding: 10px; text-align: left;">1</td>
        </tr>
        <tr>
            <td style="border: 1px solid black; padding: 10px; text-align: left;"><i>Cloze (CL)</i></td>
            <td style="border: 1px solid black; padding: 10px; text-align: left;">Text with missing words to be filled in, can be open-ended or closed-ended.</td>
            <td style="border: 1px solid black; padding: 10px; text-align: left;">3 (0.7%)</td>
            <td style="border: 1px solid black; padding: 10px; text-align: left;">6</td>
        </tr>
    </tbody>
</table>
</il>

</ul>


"""

EVALUATION_QUEUE_TEXT = """
## Evaluation Form

Please submit your model for evaluation by filling out the form located [here](https://forms.gle/4d6FYudxCVoTvj1W9).

## Some good practices before submitting a model

### 1) Make sure you can load your model and tokenizer using AutoClasses:
```python
from transformers import AutoConfig, AutoModel, AutoTokenizer
config = AutoConfig.from_pretrained("your model name", revision=revision)
model = AutoModel.from_pretrained("your model name", revision=revision)
tokenizer = AutoTokenizer.from_pretrained("your model name", revision=revision)
```
If this step fails, follow the error messages to debug your model before submitting it. It's likely your model has been improperly uploaded.

Note: make sure your model is public!
Note: if your model needs `use_remote_code=True`, we do not support this option yet but we are working on adding it, stay posted!

### 2) Convert your model weights to [safetensors](https://huggingface.co/docs/safetensors/index)
It's a new format for storing weights which is safer and faster to load and use. It will also allow us to add the number of parameters of your model to the `Extended Viewer`!

### 3) Make sure your model has an open license!
This is a leaderboard for Open LLMs, and we'd love for as many people as possible to know they can use your model πŸ€—
"""

CITATION_BUTTON_LABEL = "Copy the following snippet to cite these results"
CITATION_BUTTON_TEXT = r"""
@misc{mercorio2024disceautdeficereevaluating,
      title={Disce aut Deficere: Evaluating LLMs Proficiency on the INVALSI Italian Benchmark}, 
      author={Fabio Mercorio and Mario Mezzanzanica and Daniele Potertì and Antonio Serino and Andrea Seveso},
      year={2024},
      eprint={2406.17535},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
      url={https://arxiv.org/abs/2406.17535}, 
}
"""

QUESTION_FORMAT_TEXT = """
Performance (accuracy %) comparison of AI models across school grades and question formats for grades 2 to 13.
Grade 2 and 5 are part of primary school, grade 6 and 8 are part of middle school, and grade 10 and 13 are part of high school.
Due to the stratification and the limited number of questions in some categories, extreme values such as 100 or 0 are more attainable in the sections with few items. 

The question formats are abbreviated as:

- _Multiple Choice (MC)_: composed of a question with several answer options, among which only one is correct.
- _Multiple Complex Choice (MCC)_: composed of input questions and multiple items to answer.
- _Unique Response (RU)_: involves open-ended questions in which there are no options or suggestions and where only one answer is considered correct (with sometimes a limited number of possible variants).
- _Short Response (RB)_: is a specific sub-type of RU questions in which the answer is short, tending to consist of a single identifying word.
- _Cloze (CL)_: can be open-ended or closed-ended. In both cases, a text with missing words is provided. The aim is to complete the text by inserting the correct missing terms.
"""

MACRO_AREA_TEXT = """"
Performance (accuracy %) comparison of AI models across semantic macro areas. 
The number of questions for each category is indicated in the table headers. 

Categories are abbreviated as: 

* _LI_: Locate and identify information within the text.
* _RM_: Reconstruct the meaning of the text, locally or globally.
* _RC_: Reflect on the content or form of the text, locally or globally, and evaluate them.
* _WF_: Word formation.
* _LS_: Lexicon and semantics.
* _MO_: Morphology.
* _SP_: Spelling.
* _SY_: Syntax.
* _TP_: Textuality and pragmatics.
"""

CHANGELOG_TEXT = """
## Changelog

### 24/07/2024
- πŸ†• New model: meta-llama/Meta-Llama-3.1-405B-Instruct
- πŸ†• New model: meta-llama/Meta-Llama-3.1-70B-Instruct
- πŸ†• New model: meta-llama/Meta-Llama-3.1-8B-Instruct
- πŸ†• New model: google/gemma-2-27b-it
- πŸ†• New model: openai/gpt-4o-mini
- πŸ†• New model: cognitivecomputations/dolphin-2.9.1-llama-3-70b
- πŸ†• New model: mistralai/Mistral-Nemo-Instruct-2407
- πŸ”„ Updated model: openai/gpt-4-turbo
- πŸ”„ Updated model: openai/gpt-4o


### 05/07/2024
- πŸ†• New model: microsoft/phi-3-medium-128k-instruct
- πŸ†• New model: microsoft/phi-3-mini-128k-instruct

### 02/07/2024
- πŸ†• New model: DeepMount00/Llama-3-8b-Ita

### 30/06/2024
- πŸ†• New model: google/gemma-2-9b-it

### 28/06/2024
- πŸ†• New model: ludocomito/Minerva_3B_Ties_1.0
- πŸ†• New model: mii-llm/maestrale-chat-v0.4-beta
- πŸ†• New model: Qwen/Qwen2-72B-Instruct
- πŸ†• New model: nvidia/Nemotron-4-340B-Instruct

### 27/06/2024
- πŸ“ˆ Added changelog.
- πŸ†• New model: sapienzanlp/modello-italia-9b
- πŸ†• New model: mixtral-8x22b-instruct
- πŸ†• New model: claude-3.5-sonnet

### 26/06/2024
- πŸš€ Initial leaderboard launch.
- πŸ“Š Added evaluation metrics for the INVALSI benchmark.
- πŸ“„ Added leaderboard introduction.
- ✍ Added INVALSI benchmark description.
- πŸ“ Added evaluation queue form.
- πŸ’¬ Added citation button.
- πŸ”Ž Added question format and macro area descriptions.
"""