Updated files and statistics.

README.md

@@ -2,43 +2,37 @@
 license: mit
 language:
 - fa
+- en
+- ar
 ---
 
 # Mana Tokenizer
 
-The Mana Tokenizer is a custom-trained SentencePiece tokenizer for Persian text, trained on a combination of the Persian Wikipedia and Ganjoor datasets. The tokenizer uses the Unigram model type, optimized for handling the unique characteristics of Persian text.
+The Mana Tokenizer is a custom-trained BPE tokenizer designed for Persian text. It is trained on a combination of huge Persian corpus. The tokenizer is built using the BPE with high character coverage to handle diverse Persian text.
 
 ## Special Tokens
 
-- **UNK Token:** `<unk>`
-- **BOS Token:** `<s>`
-- **EOS Token:** `</s>`
-- **PAD Token:** `<pad>`
+- **user Token:** `<|user|>`
+- **assistant Token:** `<|assistant|>`
+- **end Token:** `<|end|>`
+- **system Token:** `<|system|>`
 
-## Usage
-
-You can load this tokenizer using the `transformers` library as follows:
-
-```python
-from transformers import PreTrainedTokenizerFast
-
-tokenizer = PreTrainedTokenizerFast.from_pretrained("tspersian/mana_tokenizer")
-
-text = "این یک تست است."
-encoded = tokenizer(text)
-print(f"Encoded: {encoded}")
-
-decoded = tokenizer.decode(encoded['input_ids'])
-print(f"Decoded: {decoded}")
-```
+## Statistics
 
+- **Model Type:** BPE
+- **Vocabulary Size:** 265,703
+- **Character Coverage:** 99.9%
+- **Total Number of Text Samples: 1,147,036
+- **Total Number of Tokens: 1,490,338
+- **Average Token Length: 4.51
+- **Corpus Size (in bytes): 1,792,210,410
 
-## Statistics
+## Training Details
 
-    Vocabulary Size: 199,997
-    Character Coverage: 99.9%
-    Total Number of Text Samples: 1,022,675
+- **Training Data: Mana Persian corpus
+- **Training Script: Mana Trainer
+- **Script Version: 1.2
 
 ## License
 
-This tokenizer is licensed under the MIT License.
+Mana tokenizer is licensed under the MIT License. 

mana_tokenizer.model → mana.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9fad0857866e56ec1708ab58165b1e536bebfe0bdaba1fbd3c82e4aeab9dd55d
-size 4663060
+oid sha256:e666a42308d210e029f8e9aa8c0056950ecd785e61514230fda35ec2962aa490
+size 2915213

mana_tokenizer.vocab → mana.vocab

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:81b98c535bc5f759cc987aeddd5dc86ff17ccde04761b245368572c21feba5ca
-size 4604696
+oid sha256:a20c30df07ce2d728dbdc5b86ba88cb65ebeca8af4361b425171051e0d3847bb
+size 11128488

special_tokens_map.json

@@ -1,6 +0,0 @@
-{
-    "unk_token": "<unk>",
-    "bos_token": "<s>",
-    "eos_token": "</s>",
-    "pad_token": "<pad>"
-}

tokenizer_config.json

@@ -1,9 +0,0 @@
-{
-    "model_type": "unigram",
-    "bos_token_id": 1,
-    "eos_token_id": 2,
-    "unk_token_id": 0,
-    "pad_token_id": 3,
-    "do_lower_case": false,
-    "max_length": 512
-}