ts-bpe-1k / special_tokens_map.json
cyrilzhang's picture
Upload tokenizer
fc51d71
raw
history blame
55 Bytes
{
"eos_token": "<|eos|>",
"pad_token": "<|pad|>"
}