|
--- |
|
language: |
|
- bn |
|
thumbnail: |
|
tags: |
|
- |
|
license: apache-2.0 |
|
datasets: |
|
- oscar |
|
- wikipedia |
|
metrics: |
|
- |
|
--- |
|
|
|
# [WIP] Albert Bengali - dev version |
|
|
|
## Model description |
|
|
|
For the moment, only the tokenizer is available. The tokenizer is based on [SentencePiece](https://github.com/google/sentencepiece) with Unigram language model segmentation algorithm. |
|
|
|
Taking into account certain characteristics of the language, we chose that: |
|
|
|
- the tokenizer passes in lower case all the texts because the Bengali language is a monocameral scrip (no difference between capital and lower case); |
|
- the sentence pieces can't go beyond the boundary of a word because the words are spaced by white spaces in the Bengali language. |
|
|
|
## Intended uses & limitations |
|
|
|
This tokenizer is adapted to the Bengali language. You can use it to pre-train an Albert model on the Bengali language. |
|
|
|
#### How to use |
|
|
|
To tokenize: |
|
|
|
```python |
|
from transformers import AlbertTokenizer |
|
tokenizer = AlbertTokenizer.from_pretrained('SaulLu/albert-bn-dev') |
|
text = "পোকেমন জাপানী ভিডিও গেম কোম্পানি নিনটেন্ডো কর্তৃক প্রকাশিত একটি মিডিয়া ফ্র্যাঞ্চাইজি।" |
|
encoded_input = tokenizer(text, return_tensors='pt') |
|
``` |
|
|
|
#### Limitations and bias |
|
|
|
Provide examples of latent issues and potential remediations. |
|
|
|
## Training data |
|
|
|
The tokenizer was trained on a random subset of 4M sentences of Bengali Oscar and Bengali Wikipedia. |
|
|
|
## Training procedure |
|
|
|
### Tokenizer |
|
|
|
The tokenizer was trained with the [SentencePiece](https://github.com/google/sentencepiece) on 8 x Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz with 16GB RAM and 36GB SWAP. |
|
|
|
``` |
|
import sentencepiece as spm |
|
config = { |
|
"input": "./dataset/oscar_bn.txt,./dataset/wikipedia_bn.txt", |
|
"input_format": "text", |
|
"model_type": "unigram", |
|
"vocab_size": 32000, |
|
"self_test_sample_size": 0, |
|
"character_coverage": 0.9995, |
|
"shuffle_input_sentence": true, |
|
"seed_sentencepiece_size": 1000000, |
|
"shrinking_factor": 0.75, |
|
"num_threads": 8, |
|
"num_sub_iterations": 2, |
|
"max_sentencepiece_length": 16, |
|
"max_sentence_length": 4192, |
|
"split_by_unicode_script": true, |
|
"split_by_number": true, |
|
"split_digits": true, |
|
"control_symbols": "[MASK]", |
|
"byte_fallback": false, |
|
"vocabulary_output_piece_score": true, |
|
"normalization_rule_name": "nmt_nfkc_cf", |
|
"add_dummy_prefix": true, |
|
"remove_extra_whitespaces": true, |
|
"hard_vocab_limit": true, |
|
"unk_id": 1, |
|
"bos_id": 2, |
|
"eos_id": 3, |
|
"pad_id": 0, |
|
"bos_piece": "[CLS]", |
|
"eos_piece": "[SEP]", |
|
"train_extremely_large_corpus": true, |
|
"split_by_whitespace": true, |
|
"model_prefix": "./spiece", |
|
"input_sentence_size": 4000000, |
|
"user_defined_symbols": "(,),-,.,–,£,।", |
|
} |
|
spm.SentencePieceTrainer.train(**config) |
|
``` |
|
|
|
<!-- ## Eval results |
|
|
|
### BibTeX entry and citation info |
|
|
|
```bibtex |
|
@inproceedings{..., |
|
year={2020} |
|
} |
|
``` --> |
|
|
