---
language:
- fa
library_name: transformers
widget:
  - text: "ز سوزناکی گفتار من [MASK] بگریست"
    example_title: "Poetry 1"
  - text: "نظر از تو برنگیرم همه [MASK] تا بمیرم که تو در دلم نشستی و سر مقام داری"
    example_title: "Poetry 2"
  - text: "هر ساعتم اندرون بجوشد [MASK] را وآگاهی نیست مردم بیرون را"
    example_title: "Poetry 3"
  - text: "غلام همت آن رند عافیت سوزم که در گدا صفتی [MASK] داند"
    example_title: "Poetry 4"
  - text: "این [MASK] اولشه."
    example_title: "Informal 1"
  - text: "دیگه خسته شدم! [MASK] اینم شد کار؟!"
    example_title: "Informal 2"
  - text: "فکر نکنم به موقع برسیم. بهتره [MASK] این یکی بشیم."
    example_title: "Informal 3"
  - text: "تا صبح بیدار موندم و داشتم برای [MASK] آماده می شدم."
    example_title: "Informal 4"
  - text: "زندگی بدون [MASK] خسته‌کننده است."
    example_title: "Formal 1"
  - text: "در حکم اولیه این شرکت مجاز به فعالیت شد ولی پس از بررسی مجدد، مجوز این شرکت [MASK] شد."
    example_title: "Formal 2"
---


# FaBERT: Pre-training BERT on Persian Blogs

## Model Details

FaBERT is a Persian BERT-base model trained on the diverse HmBlogs corpus, encompassing both casual and formal Persian texts. Developed for natural language processing tasks, FaBERT is a robust solution for processing Persian text. Through evaluation across various Natural Language Understanding (NLU) tasks, FaBERT consistently demonstrates notable improvements, while having a compact model size. Now available on Hugging Face, integrating FaBERT into your projects is hassle-free. Experience enhanced performance without added complexity as FaBERT tackles a variety of NLP tasks.

## Features
- Pre-trained on the diverse HmBlogs corpus consisting more than 50 GB of text from Persian Blogs
- Remarkable performance across various downstream NLP tasks
- BERT architecture with 124 million parameters

## Useful Links
- **Repository:** [FaBERT on Github](https://github.com/SBU-NLP-LAB/FaBERT)
- **Paper:** [arXiv preprint](https://arxiv.org/abs/2402.06617)

## Usage

### Loading the Model with MLM head

```python
from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("sbunlp/fabert") # make sure to use the default fast tokenizer
model = AutoModelForMaskedLM.from_pretrained("sbunlp/fabert")
```
### Downstream Tasks

Similar to the original English BERT, FaBERT can be fine-tuned on many downstream tasks.(https://huggingface.co/docs/transformers/en/training) 

Examples on Persian datasets are available in our [GitHub repository](#useful-links).

**make sure to use the default Fast Tokenizer**

## Training Details

FaBERT was pre-trained with the MLM (WWM) objective, and the resulting perplexity on validation set was 7.76.

| Hyperparameter    | Value        |
|-------------------|:--------------:|
| Batch Size        | 32           |
| Optimizer         | Adam         |
| Learning Rate     | 6e-5         |
| Weight Decay      | 0.01         |
| Total Steps       | 18 Million    |
| Warmup Steps      | 1.8 Million   |
| Precision Format  | TF32          |

## Evaluation

Here are some key performance results for the FaBERT model:

**Sentiment Analysis**
| Task         | FaBERT | ParsBERT | XLM-R |
|:-------------|:------:|:--------:|:-----:|
| MirasOpinion | **87.51**      | 86.73     | 84.92  |
| MirasIrony | 74.82      | 71.08     | **75.51**  |
| DeepSentiPers | **79.85**      | 74.94     | 79.00  |

**Named Entity Recognition**
| Task         | FaBERT | ParsBERT | XLM-R |
|:-------------|:------:|:--------:|:-----:|
| PEYMA        |   **91.39**    |   91.24   | 90.91  |
| ParsTwiner   |   **82.22**    |  81.13   | 79.50  |
| MultiCoNER v2   |   57.92    |   **58.09**   | 51.47  |

**Question Answering**
| Task         | FaBERT | ParsBERT | XLM-R |
|:-------------|:------:|:--------:|:-----:|
| ParsiNLU | **55.87**      | 44.89     | 42.55  |
| PQuAD  | 87.34      | 86.89     | **87.60**  |
| PCoQA  | **53.51**      | 50.96     | 51.12  |

**Natural Language Inference & QQP**
| Task         | FaBERT | ParsBERT | XLM-R |
|:-------------|:------:|:--------:|:-----:|
| FarsTail | **84.45**      | 82.52     | 83.50  |
| SBU-NLI | **66.65**      | 58.41     | 58.85  |
| ParsiNLU QQP | **82.62**      | 77.60     | 79.74  |

**Number of Parameters**
|          | FaBERT | ParsBERT | XLM-R |
|:-------------|:------:|:--------:|:-----:|
| Parameter Count (M) | 124      | 162     | 278  |
| Vocabulary Size (K) | 50      | 100     | 250  |

For a more detailed performance analysis refer to the paper.

## How to Cite

If you use FaBERT in your research or projects, please cite it using the following BibTeX:

```bibtex
@article{masumi2024fabert,
  title={FaBERT: Pre-training BERT on Persian Blogs},
  author={Masumi, Mostafa and Majd, Seyed Soroush and Shamsfard, Mehrnoush and Beigy, Hamid},
  journal={arXiv preprint arXiv:2402.06617},
  year={2024}
}
```