redditbot / added_tokens.json
lagodw's picture
add tokenizer
1d50d0b
raw
history blame contribute delete
93 Bytes
{"<|startoftext|>": 50257, "<sep>": 50258, "<|pad|>": 50259, "<cls>": 50260, "<mask>": 50261}