from dataclasses import dataclass from enum import Enum @dataclass class Task: benchmark: str metric: str col_name: str # Select your tasks here # --------------------------------------------------- class Tasks(Enum): # task_key in the json file, metric_key in the json file, name to display in the leaderboard task0 = Task("main", "acc", "Overall") NUM_FEWSHOT = 0 # Change with your few shot # --------------------------------------------------- # Your leaderboard name TITLE = """

👩‍🏫Invalsi Leaderboard

""" # What does your leaderboard evaluate? INTRODUCTION_TEXT = """ Welcome into CRISP Bicocca Invalsi Leaderboard page! We adapted the INVALSI benchmark for automated LLM evaluation, which involves rigorous adaptation of the test format to suit automated processing while retaining the essence of the original tests. In this leaderboard, we provide a detailed assessment of current LLMs, offering a crucial reference point for the academic community. Researchers are invited to submit their models for ongoing evaluation, ensuring the benchmark remains a current and valuable resource. For more information on the benchmark, please refer to our arXiv paper here and read the "About" section. """ # Which evaluations are you running? how can people reproduce what you have? LLM_BENCHMARKS_TEXT = f""" ## INVALSI Benchmark The INVALSI (National Institute for the Evaluation of the Education and Training System) test has been essential to Italy's educational assessment framework since the 2005-2006 academic year. It aims to evaluate students' competencies across various subjects, including Italian language and mathematics, at multiple stages of the education system in a standardised manner. The test's primary objective is to assess students' linguistic proficiency, a crucial skill that schools are tasked with developing as per national educational guidelines. Since the test covers a wide range of linguistic and comprehension skills, using it to evaluate LLMs can provide a detailed view of a model's proficiency in handling real-world, nuanced language tasks designed for human learners. The test's structured and standardised nature makes it an excellent benchmark for comparing different LLMs with questions culturally and contextually relevant to Italian speakers. However, our findings are relevant across all languages since they assess various general capabilities, such as word formation and text comprehension abilities. Also, since it is designed for different educational stages, it provides varied levels of complexity and challenge. This aspect can gauge an LLM's capability at different difficulty levels, reflecting its potential scalability and adaptability across simpler to more complex linguistic tasks. ### Data Collection We have collected from public sources 58 unique tests, divided into 141 unique units, with 2114 questions and 2808 unique items. Some questions are subdivided into multiple items, each requiring a specific answer. The data has been gathered from the Gestinv database. Questions' formatting is sometimes not adequately structured for LLM evaluation; for instance, it is sometimes impossible to automatically transcribe the questions into structured fields, necessitating further inspection of images and PDFs. For this reason, we also collected corresponding PDF files and images. Manual inspection was required to ensure accuracy. In cases where questions involved graphical elements, we modified them into an appropriate multiple-choice format. For example, if the task required a student to find and underscore a word, we reformulated the question to allow selection from multiple choices. Similarly, if the task involved drawing a line between two groups of concepts—a common task for younger students—we rephrased it to involve choosing the correct association from given options. Generally, we aimed to adapt the questions to a format that allows the model to select the correct answer from a pool of choices if it aligns with the original type of question. ### Dataset Characteristics """ EVALUATION_QUEUE_TEXT = """ ## Evaluation Form Please submit your model for evaluation by filling out the form located [here](https://forms.gle/tkbJME4SS9LF8XTK7). ## Some good practices before submitting a model ### 1) Make sure you can load your model and tokenizer using AutoClasses: ```python from transformers import AutoConfig, AutoModel, AutoTokenizer config = AutoConfig.from_pretrained("your model name", revision=revision) model = AutoModel.from_pretrained("your model name", revision=revision) tokenizer = AutoTokenizer.from_pretrained("your model name", revision=revision) ``` If this step fails, follow the error messages to debug your model before submitting it. It's likely your model has been improperly uploaded. Note: make sure your model is public! Note: if your model needs `use_remote_code=True`, we do not support this option yet but we are working on adding it, stay posted! ### 2) Convert your model weights to [safetensors](https://huggingface.co/docs/safetensors/index) It's a new format for storing weights which is safer and faster to load and use. It will also allow us to add the number of parameters of your model to the `Extended Viewer`! ### 3) Make sure your model has an open license! This is a leaderboard for Open LLMs, and we'd love for as many people as possible to know they can use your model 🤗 """ CITATION_BUTTON_LABEL = "Copy the following snippet to cite these results" CITATION_BUTTON_TEXT = r""" """ QUESTION_FORMAT_TEXT = """ Question Format evaluation """ MACRO_AREA_TEXT = """" This table shows the evaluation of the models by macro area. The evaluation is based on the following metrics: """