README.md

# This is only a tokenizer.

- This tokenizer is a PreTrainedTokenizerFast which is trained on raygx/Nepali-Extended-Corpus datasets.
- This tokenizer is trained from scratch using Tokenizers library.
- This tokenizer uses
  - Model: BPE(unk_token="[UNK]")
  - Normalizer: normalizers.Sequence([NFD(),Strip()])
  - Pre-processor: pre_tokenizers.Sequence([Whitespace(),Digits(individual_digits=True), Punctuation()])
  - Post-processor: BertProcessing

[Code is available here.](https://www.kaggle.com/code/reganmaharjan/nepali-tokenizers-4-transformers/)