# EsperBERTo Model Card ## Model Description EsperBERTo is a RoBERTa-like model specifically trained from scratch on the Esperanto language using a large corpus from the OSCAR and Leipzig Corpora Collection. It is designed to perform masked language modeling and other text-based prediction tasks. This model is ideal for understanding and generating Esperanto text. ### Datasets - **OSCAR Corpus (Esperanto)**: Extracted from Common Crawl dumps, filtered by language classification. - **Leipzig Corpora Collection (Esperanto)**: Includes texts from news, literature, and Wikipedia. ### Preprocessing - Trained a byte-level Byte-pair encoding tokenizer with a vocabulary size of 52,000 tokens. ### Hyperparameters - **Number of Epochs**: 1 - **Batch Size per GPU**: 64 - **Training Steps for Saving**: 10,000 - **Limit of Saved Models**: 2 - **Loss Calculation**: Prediction loss only ### Software and Libraries - **Transformers Library Version**: [Transformers](https://github.com/huggingface/transformers) - **Training Script**: `run_language_modeling.py` ```python from transformers import pipeline fill_mask = pipeline( "fill-mask", model="SamJoshua/EsperBERTo-small", tokenizer="SamJoshua/EsperBERTo-small" ) fill_mask("Jen la komenco de bela .") ``` ## Evaluation Results The model has not yet been evaluated on a standardized test set. Future updates will include evaluation metrics such as perplexity and accuracy on a held-out validation set. ## Intended Uses & Limitations **Intended Uses**: This model is intended for researchers, developers, and language enthusiasts who wish to explore Esperanto language processing for tasks like text generation, sentiment analysis, and more. **Limitations**: - The model is trained only for one epoch due to computational constraints, which may affect its understanding of more complex language structures. - As the model is trained on public web text, it may inadvertently learn and replicate social biases present in the training data. Feel free to contribute to the model by fine-tuning on specific tasks or extending its training with more data or epochs. This model serves as a baseline for further research and development in Esperanto language modeling.