--- license: cc-by-sa-4.0 language: - hr - bs - sr datasets: - classla/xlm-r-bertic-data --- # XLM-R-BERTić This model was produced by pre-training [XLM-Roberta-large](https://huggingface.co/xlm-roberta-large) 48k steps on South Slavic languages using [XLM-R-BERTić dataset](https://huggingface.co/datasets/classla/xlm-r-bertic-data) # Benchmarking Three tasks were chosen for model evaluation: * Named Entity Recognition (NER) * Sentiment regression * COPA (Choice of plausible alternatives) In all cases, this model was finetuned for specific downstream tasks. ## NER Average macro-F1 scores from three runs were used to evaluate performance. Datasets used: [hr500k](https://huggingface.co/datasets/classla/hr500k), [ReLDI-sr](https://huggingface.co/datasets/classla/reldi_sr), [ReLDI-hr](https://huggingface.co/datasets/classla/reldi_hr), and [SETimes.SR](https://huggingface.co/datasets/classla/setimes_sr). | system | dataset | F1 score | |:-----------------------------------------------------------------------|:--------|---------:| | [**XLM-R-BERTić**](https://huggingface.co/classla/xlm-r-bertic) | hr500k | 0.927 | | [BERTić](https://huggingface.co/classla/bcms-bertic) | hr500k | 0.925 | | [XLM-R-SloBERTić](https://huggingface.co/classla/xlm-r-slobertic) | hr500k | 0.923 | | [XLM-Roberta-Large](https://huggingface.co/xlm-roberta-large) | hr500k | 0.919 | | [crosloengual-bert](https://huggingface.co/EMBEDDIA/crosloengual-bert) | hr500k | 0.918 | | [XLM-Roberta-Base](https://huggingface.co/xlm-roberta-base) | hr500k | 0.903 | | system | dataset | F1 score | |:-----------------------------------------------------------------------|:---------|---------:| | [XLM-R-SloBERTić](https://huggingface.co/classla/xlm-r-slobertic) | ReLDI-hr | 0.812 | | [**XLM-R-BERTić**](https://huggingface.co/classla/xlm-r-bertic) | ReLDI-hr | 0.809 | | [crosloengual-bert](https://huggingface.co/EMBEDDIA/crosloengual-bert) | ReLDI-hr | 0.794 | | [BERTić](https://huggingface.co/classla/bcms-bertic) | ReLDI-hr | 0.792 | | [XLM-Roberta-Large](https://huggingface.co/xlm-roberta-large) | ReLDI-hr | 0.791 | | [XLM-Roberta-Base](https://huggingface.co/xlm-roberta-base) | ReLDI-hr | 0.763 | | system | dataset | F1 score | |:-----------------------------------------------------------------------|:-----------|---------:| | [XLM-R-SloBERTić](https://huggingface.co/classla/xlm-r-slobertic) | SETimes.SR | 0.949 | | [**XLM-R-BERTić**](https://huggingface.co/classla/xlm-r-bertic) | SETimes.SR | 0.940 | | [BERTić](https://huggingface.co/classla/bcms-bertic) | SETimes.SR | 0.936 | | [XLM-Roberta-Large](https://huggingface.co/xlm-roberta-large) | SETimes.SR | 0.933 | | [crosloengual-bert](https://huggingface.co/EMBEDDIA/crosloengual-bert) | SETimes.SR | 0.922 | | [XLM-Roberta-Base](https://huggingface.co/xlm-roberta-base) | SETimes.SR | 0.914 | | system | dataset | F1 score | |:-----------------------------------------------------------------------|:---------|---------:| | [**XLM-R-BERTić**](https://huggingface.co/classla/xlm-r-bertic) | ReLDI-sr | 0.841 | | [XLM-R-SloBERTić](https://huggingface.co/classla/xlm-r-slobertic) | ReLDI-sr | 0.824 | | [BERTić](https://huggingface.co/classla/bcms-bertic) | ReLDI-sr | 0.798 | | [XLM-Roberta-Large](https://huggingface.co/xlm-roberta-large) | ReLDI-sr | 0.774 | | [crosloengual-bert](https://huggingface.co/EMBEDDIA/crosloengual-bert) | ReLDI-sr | 0.751 | | [XLM-Roberta-Base](https://huggingface.co/xlm-roberta-base) | ReLDI-sr | 0.734 | ## Sentiment regression [ParlaSent dataset](https://huggingface.co/datasets/classla/ParlaSent) was used to evaluate sentiment regression for Bosnian, Croatian, and Serbian languages. The procedure is explained in greater detail in the dedicated [benchmarking repository](https://github.com/clarinsi/benchich/tree/main/sentiment). | system | train | test | r^2 | |:-----------------------------------------------------------------------|:--------------------|:-------------------------|------:| | [xlm-r-parlasent](https://huggingface.co/classla/xlm-r-parlasent) | ParlaSent_BCS.jsonl | ParlaSent_BCS_test.jsonl | 0.615 | | [BERTić](https://huggingface.co/classla/bcms-bertic) | ParlaSent_BCS.jsonl | ParlaSent_BCS_test.jsonl | 0.612 | | [XLM-R-SloBERTić](https://huggingface.co/classla/xlm-r-slobertic) | ParlaSent_BCS.jsonl | ParlaSent_BCS_test.jsonl | 0.607 | | [XLM-Roberta-Large](https://huggingface.co/xlm-roberta-large) | ParlaSent_BCS.jsonl | ParlaSent_BCS_test.jsonl | 0.605 | | [**XLM-R-BERTić**](https://huggingface.co/classla/xlm-r-bertic) | ParlaSent_BCS.jsonl | ParlaSent_BCS_test.jsonl | 0.601 | | [crosloengual-bert](https://huggingface.co/EMBEDDIA/crosloengual-bert) | ParlaSent_BCS.jsonl | ParlaSent_BCS_test.jsonl | 0.537 | | [XLM-Roberta-Base](https://huggingface.co/xlm-roberta-base) | ParlaSent_BCS.jsonl | ParlaSent_BCS_test.jsonl | 0.500 | | dummy (mean) | ParlaSent_BCS.jsonl | ParlaSent_BCS_test.jsonl | -0.12 | ## COPA | system | dataset | Accuracy score | |:-----------------------------------------------------------------------|:--------|---------------:| | [BERTić](https://huggingface.co/classla/bcms-bertic) | Copa-SR | 0.689 | | [XLM-R-SloBERTić](https://huggingface.co/classla/xlm-r-slobertic) | Copa-SR | 0.665 | | [**XLM-R-BERTić**](https://huggingface.co/classla/xlm-r-bertic) | Copa-SR | 0.637 | | [crosloengual-bert](https://huggingface.co/EMBEDDIA/crosloengual-bert) | Copa-SR | 0.607 | | [XLM-Roberta-Base](https://huggingface.co/xlm-roberta-base) | Copa-SR | 0.573 | | [XLM-Roberta-Large](https://huggingface.co/xlm-roberta-large) | Copa-SR | 0.570 | | system | dataset | Accuracy score | |:-----------------------------------------------------------------------|:--------|---------------:| | [BERTić](https://huggingface.co/classla/bcms-bertic) | Copa-HR | 0.669 | | [XLM-R-SloBERTić](https://huggingface.co/classla/xlm-r-slobertic) | Copa-HR | 0.628 | | [**XLM-R-BERTić**](https://huggingface.co/classla/xlm-r-bertic) | Copa-HR | 0.635 | | [crosloengual-bert](https://huggingface.co/EMBEDDIA/crosloengual-bert) | Copa-HR | 0.669 | | [XLM-Roberta-Base](https://huggingface.co/xlm-roberta-base) | Copa-HR | 0.585 | | [XLM-Roberta-Large](https://huggingface.co/xlm-roberta-large) | Copa-HR | 0.571 | # Citation Please cite the following paper: ``` @inproceedings{ljubesic-etal-2024-language, title = "Language Models on a Diet: Cost-Efficient Development of Encoders for Closely-Related Languages via Additional Pretraining", author = "Ljube{\v{s}}i{\'c}, Nikola and Suchomel, V{\'\i}t and Rupnik, Peter and Kuzman, Taja and van Noord, Rik", editor = "Melero, Maite and Sakti, Sakriani and Soria, Claudia", booktitle = "Proceedings of the 3rd Annual Meeting of the Special Interest Group on Under-resourced Languages @ LREC-COLING 2024", month = may, year = "2024", address = "Torino, Italia", publisher = "ELRA and ICCL", url = "https://aclanthology.org/2024.sigul-1.23", pages = "189--203", } ```