Hanbaike
/

kyrgyz_unigram_24000

Model card Files Files and versions

Hanbaike commited on May 8

Commit

50ae90f

·

verified ·

1 Parent(s): 5cd0ca4

Upload tokenizer_config.json with huggingface_hub

Files changed (1) hide show

tokenizer_config.json +2 -1

tokenizer_config.json CHANGED Viewed

@@ -4,5 +4,6 @@
   "bos_token": "<s>",
   "eos_token": "</s>",
   "pad_token": "<pad>",
-  "tokenizer_class": "PreTrainedTokenizerFast"
 }

   "bos_token": "<s>",
   "eos_token": "</s>",
   "pad_token": "<pad>",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "tokenizer_file": "kyrgyz_unigram_24000.model"
 }