|
--- |
|
language: |
|
- en |
|
- ja |
|
license: mit |
|
datasets: |
|
- snow_simplified_japanese_corpus |
|
tags: |
|
- ja |
|
- japanese |
|
- tokenizer |
|
widget: |
|
- text: "誰が一番に着くか私には分かりません。" |
|
--- |
|
|
|
# Japanese Dummy Tokenizer |
|
|
|
Repository containing a dummy Japanese Tokenizer trained on ```snow_simplified_japanese_corpus``` dataset. The tokenizer has been trained using Hugging Face datasets in a streaming manner. |
|
|
|
## Intended uses & limitations |
|
|
|
You can use this tokenizer to tokenize Japanese sentences. |
|
|
|
## How to use it |
|
|
|
``` |
|
from transformers import AutoTokenizer |
|
|
|
tokenizer = AutoTokenizer.from_pretrained("ybelkada/japanese-dummy-tokenizer") |
|
``` |
|
|
|
## How to train the tokenizer |
|
|
|
Check the file ```tokenizer.py```, you can freely adapt it to other datasets. This tokenizer is based on the tokenizer from ```csebuetnlp/mT5_multilingual_XLSum```. |
