|
--- |
|
language: ar |
|
datasets: |
|
- oscar |
|
- wikipedia |
|
tags: |
|
- ar |
|
- masked-lm |
|
--- |
|
|
|
|
|
# Arabic-ALBERT Xlarge |
|
|
|
Arabic edition of ALBERT Xlarge pretrained language model |
|
|
|
_If you use any of these models in your work, please cite this work as:_ |
|
|
|
``` |
|
@software{ali_safaya_2020_4718724, |
|
author = {Ali Safaya}, |
|
title = {Arabic-ALBERT}, |
|
month = aug, |
|
year = 2020, |
|
publisher = {Zenodo}, |
|
version = {1.0.0}, |
|
doi = {10.5281/zenodo.4718724}, |
|
url = {https://doi.org/10.5281/zenodo.4718724} |
|
} |
|
``` |
|
|
|
## Pretraining data |
|
|
|
The models were pretrained on ~4.4 Billion words: |
|
|
|
- Arabic version of [OSCAR](https://oscar-corpus.com/) (unshuffled version of the corpus) - filtered from [Common Crawl](http://commoncrawl.org/) |
|
- Recent dump of Arabic [Wikipedia](https://dumps.wikimedia.org/backup-index.html) |
|
|
|
__Notes on training data:__ |
|
|
|
- Our final version of corpus contains some non-Arabic words inlines, which we did not remove from sentences since that would affect some tasks like NER. |
|
- Although non-Arabic characters were lowered as a preprocessing step, since Arabic characters do not have upper or lower case, there is no cased and uncased version of the model. |
|
- The corpus and vocabulary set are not restricted to Modern Standard Arabic, they contain some dialectical Arabic too. |
|
|
|
## Pretraining details |
|
|
|
- These models were trained using Google ALBERT's github [repository](https://github.com/google-research/albert) on a single TPU v3-8 provided for free from [TFRC](https://www.tensorflow.org/tfrc). |
|
- Our pretraining procedure follows training settings of bert with some changes: trained for 7M training steps with batchsize of 64, instead of 125K with batchsize of 4096. |
|
|
|
## Models |
|
|
|
| | albert-base | albert-large | albert-xlarge | |
|
|:---:|:---:|:---:|:---:| |
|
| Hidden Layers | 12 | 24 | 24 | |
|
| Attention heads | 12 | 16 | 32 | |
|
| Hidden size | 768 | 1024 | 2048 | |
|
|
|
## Results |
|
|
|
For further details on the models performance or any other queries, please refer to [Arabic-ALBERT](https://github.com/KUIS-AI-Lab/Arabic-ALBERT/) |
|
|
|
## How to use |
|
|
|
You can use these models by installing `torch` or `tensorflow` and Huggingface library `transformers`. And you can use it directly by initializing it like this: |
|
|
|
```python |
|
|
|
from transformers import AutoTokenizer, AutoModel |
|
|
|
# loading the tokenizer |
|
tokenizer = AutoTokenizer.from_pretrained("kuisailab/albert-xlarge-arabic") |
|
|
|
# loading the model |
|
model = AutoModelForMaskedLM.from_pretrained("kuisailab/albert-xlarge-arabic") |
|
|
|
``` |
|
|
|
## Acknowledgement |
|
|
|
Thanks to Google for providing free TPU for the training process and for Huggingface for hosting these models on their servers 😊 |
|
|
