license: mit
language:
- ar
pipeline_tag: text-generation
tags:
- 'arabic '
- text-generation
Model Description
- Model Name: ArabianGPT
- Architecture: GPT-2
- Layers: 12
- Model Size: 134M
- Context Window Size: 768
[! NOTE] ArabianGPT is a custom-trained version of the GPT-2 base model, specifically tailored for the Arabic language. It is designed to understand and generate Arabic text, making it suitable for various natural language processing tasks in Arabic.
Training
- Dataset: Abu Elkhiar Corpus
- Size: 15.5 GB
- Number of Words: 237,814,541
- Number of Tokens: 1,752,421,071
- Number of Parameters : 134 M Params
- Steps: 337,500
- Loss: 3.97
The model was trained on the Abu Elkhiar dataset, a comprehensive Arabic text corpus encompassing a wide range of topics. The training process focused on adapting the model to understand the nuances and complexities of the Arabic language.
Tokenizer
Type: Custom trained SentencePiece tokenizer Vocabulary Size: 64K
We employed AraNizer, a custom trained tokenizer based on the SentencePiece model, with a vocabulary size of 64K. This choice was made to optimize the model's performance for the specific characteristics of the Arabic language.
More info about AraNizer can be found here Link
Usage
ArabianGPT can be used for text generation tasks in Arabic.
How to use
Here is how to use this model to generate ruby function documentation using Transformers SummarizationPipeline:
from transformers import pipeline
pipe = pipeline("text-generation", model="riotu-lab/ArabianGPT-base" , max_new_tokens = 512)
text = ''
pipe.predict(text)
Limitations
As with any language model, ArabianGPT may have limitations in understanding context or generating text in certain scenarios. Users should be aware of these limitations and use the model accordingly.
Ethical Considerations
We emphasize responsible usage of ArabianGPT. Users should ensure that the generated text is used ethically and does not propagate misinformation or harmful content.
Citation
If you use ArabianGPT in your research or application, please cite it as follows:
@misc{ArabianGPT, 2023,
title={ArabianGPT: A GPT-2 Based Language Model for Arabic},
author={Najar, Omar and Sibaee, Serry and Ghouti, Lahouari and Koubaa, Anis},
affiliation={Prince Sultan University, Riyadh, Saudi Arabia},
year={2023},
}
Acknowledgments
We thank Prince Sultan University, especially the Robotics and Internet of Things Lab, for their support.
Contact
For inquiries regarding ArabianGPT, please contact onajar@psu.edu.sa.