update readme.md
Browse files
README.md
CHANGED
|
@@ -21,6 +21,15 @@ tags:
|
|
| 21 |
- **Fertility Score**: 1.691
|
| 22 |
- It supports Arabic Diacritization
|
| 23 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 24 |
## How to Use the Aranizer Tokenizer
|
| 25 |
|
| 26 |
The Aranizer tokenizer can be easily loaded using the `transformers` library from Hugging Face. Below is an example of how to load and use the tokenizer in your Python project:
|
|
|
|
| 21 |
- **Fertility Score**: 1.691
|
| 22 |
- It supports Arabic Diacritization
|
| 23 |
|
| 24 |
+
## Aranizer Collection Achieved State of the Art Arabic Tokenizer
|
| 25 |
+
|
| 26 |
+
The Aranizer tokenizer has achieved state-of-the-art results on the [Arabic Tokenizers Leaderboard](https://huggingface.co/spaces/MohamedRashad/arabic-tokenizers-leaderboard) on Hugging Face. Below is a screenshot highlighting this achievement:
|
| 27 |
+
|
| 28 |
+
<a href="https://ibb.co/L6rwnb6">
|
| 29 |
+
<img src="https://i.ibb.co/q9CGkX9/Screenshot-2024-08-25-at-10-20-56-PM.png" alt="Screenshot showing the Aranizer Tokenizer achieving state of the art" border="0">
|
| 30 |
+
</a>
|
| 31 |
+
|
| 32 |
+
|
| 33 |
## How to Use the Aranizer Tokenizer
|
| 34 |
|
| 35 |
The Aranizer tokenizer can be easily loaded using the `transformers` library from Hugging Face. Below is an example of how to load and use the tokenizer in your Python project:
|