XLM Roberta Tokenizer trained with 162M tokens of Khmer text.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("seanghay/xlm-roberta-khmer-32k-tokenizer")
tokenizer.tokenize("សួស្ដីកម្ពុជា!")

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model is not currently available via any of the supported third-party Inference Providers, and HF Inference API was unable to determine this model’s pipeline type.