|
--- |
|
license: mit |
|
language: |
|
- ar |
|
pipeline_tag: text-generation |
|
tags: |
|
- 'arabic ' |
|
- text-generation |
|
--- |
|
# Model Description |
|
|
|
* Model Name: ArabianGPT |
|
* Architecture: GPT-2 |
|
* Layers: 12 |
|
* Model Size: 134M |
|
* Context Window Size: 768 |
|
|
|
> [! NOTE] |
|
> ArabianGPT is a custom-trained version of the GPT-2 base model, specifically tailored for the Arabic language. It is designed to understand and generate Arabic text, making it suitable for various natural language processing tasks in Arabic. |
|
|
|
# Training |
|
* Dataset: Abu Elkhiar Corpus |
|
* Size: 15.5 GB |
|
* Number of Words: 237,814,541 |
|
* Number of Tokens: 1,752,421,071 |
|
* Number of Parameters : 134 M Params |
|
* Steps: 337,500 |
|
* Loss: 3.97 |
|
|
|
> [!NOTE] |
|
> The model was trained on the Abu Elkhiar dataset, a comprehensive Arabic text corpus encompassing a wide range of topics. The training process focused on adapting the model to understand the nuances and complexities of the Arabic language. |
|
|
|
# Tokenizer |
|
Type: Custom trained SentencePiece tokenizer |
|
Vocabulary Size: 64K |
|
|
|
> We employed AraNizer, a custom trained tokenizer based on the SentencePiece model, with a vocabulary size of 64K. This choice was made to optimize the model's performance for the specific characteristics of the Arabic language. |
|
|
|
More info about AraNizer can be found here [Link](https://github.com/omarnj-lab/aranizer/tree/main) |
|
|
|
|
|
# Usage |
|
ArabianGPT can be used for text generation tasks in Arabic. |
|
|
|
### How to use |
|
|
|
Here is how to use this model to generate ruby function documentation using Transformers SummarizationPipeline: |
|
|
|
```python |
|
from transformers import pipeline |
|
|
|
pipe = pipeline("text-generation", model="riotu-lab/ArabianGPT-base" , max_new_tokens = 512) |
|
|
|
text = '' |
|
|
|
pipe.predict(text) |
|
``` |
|
|
|
# Limitations |
|
> [!TIP] |
|
> As with any language model, ArabianGPT may have limitations in understanding context or generating text in certain scenarios. Users should be aware of these limitations and use the model accordingly. |
|
|
|
# Ethical Considerations |
|
We emphasize responsible usage of ArabianGPT. Users should ensure that the generated text is used ethically and does not propagate misinformation or harmful content. |
|
|
|
# Citation |
|
|
|
If you use ArabianGPT in your research or application, please cite it as follows: |
|
|
|
``` |
|
@misc{ArabianGPT, 2023, |
|
title={ArabianGPT: A GPT-2 Based Language Model for Arabic}, |
|
author={Najar, Omar and Sibaee, Serry and Ghouti, Lahouari and Koubaa, Anis}, |
|
affiliation={Prince Sultan University, Riyadh, Saudi Arabia}, |
|
year={2023}, |
|
} |
|
``` |
|
# Acknowledgments |
|
> We thank Prince Sultan University, especially the Robotics and Internet of Things Lab, for their support. |
|
|
|
# Contact |
|
For inquiries regarding ArabianGPT, please contact onajar@psu.edu.sa. |
