Spaces:
Runtime error
Runtime error
| """ | |
| This file contains the text content for the leaderboard client. | |
| """ | |
| HEADER_MARKDOWN = """ | |
| # 🇨🇿 BenCzechMark [Beta Preview] | |
| Welcome to the leaderboard! | |
| Here you can compare models on tasks in Czech language and/or submit your own model. We use our modified fork of [lm-evaluation-harness](https://github.com/DCGM/lm-evaluation-harness) to evaluate every model under same protocol. | |
| - Head to submission page to learn about submission details. | |
| - See about page for brief description of our evaluation protocol & win score mechanism, citation information, and future directions for this benchmark. | |
| """ | |
| LEADERBOARD_TAB_TITLE_MARKDOWN = """ | |
| ## Leaderboard | |
| """ | |
| SUBMISSION_TAB_TITLE_MARKDOWN = """ | |
| ## How to submit | |
| 1. Head down to our modified fork of [lm-evaluation-harness](https://github.com/DCGM/lm-evaluation-harness). | |
| Follow the instructions and evaluate your model on all 🇨🇿 BenCzechMark tasks, while logging your lm harness outputs into designated folder. | |
| 2. Use our script [compile_log_files.py](https://huggingface.co/spaces/CZLC/BenCzechMark/blob/main/compile_log_files.py) for processing log files from your designated folder into single compact submission file that contains everything we need. | |
| Example usage: | |
| - Download sample outputs for csmpt7b from [csmpt_logdir.zip](https://czechllm.fit.vutbr.cz/csmpt7b/sample_results/csmpt_logdir.zip). | |
| - Unzip. | |
| - Run the script as with python (with libs jsonlines and tqdm) | |
| ```bash | |
| python compile_log_files.py \ | |
| -i "<your_local_path_to_folder>/csmpt_logdir/csmpt/eval_csmpt7b*" \ | |
| -o "<your_local_path_to_outfolder>/sample_submission.json" | |
| ``` | |
| 3. Upload your file, and fill the form below! | |
| ## Submission | |
| To submit your model, please fill in the form below. | |
| - *Team name:* The name of your team, as it will appear on the leaderboard | |
| - *Model name:* The name of your model | |
| - *Model type:* The type of your model (chat, pretrained, ensemble) | |
| - *Parameters (B):* The number of parameters of your model in billions (10⁹) | |
| - *Input length (# tokens):* The number of input tokens that led to the results | |
| - *Precision:* The precision with which the results were obtained | |
| - *Description:* Short description of your submission (optional) | |
| - *Link to model:* Link to the model's repository or documentation | |
| - *Upload your results:* Results json file to submit | |
| After filling in the form, click the **Pre-submit model** button. | |
| This will run a comparison of your model with the existing leaderboard models. | |
| After the tournament is complete, you will be able to submit your model to the leaderboard. | |
| """ | |
| RANKING_AFTER_SUBMISSION_MARKDOWN = """ | |
| This is how will ranking look like after your submission: | |
| """ | |
| SUBMISSION_DETAILS_MARKDOWN = """ | |
| Do you really want to submit a model? This action is irreversible. | |
| """ | |
| MORE_DETAILS_MARKDOWN = """ | |
| ## Model details: | |
| """ | |
| MODAL_SUBMIT_MARKDOWN = """ | |
| Are you sure you want to submit your model? | |
| """ | |