polyglot-ko-3.8b / tokenizer_config.json
jason9693's picture
Upload with huggingface_hub
1b01a45
raw
history blame
90 Bytes
{"pad_token": "<pad>", "eos_token": "<eos>", "tokenizer_class": "PreTrainedTokenizerFast"}