screeve-lemmatizer / special_tokens_map.json
Nargizi's picture
add tokenizer
a651a9b
raw
history blame
127 Bytes
{
"cls_token": "<START>",
"mask_token": "[MASK]",
"pad_token": "<PAD>",
"sep_token": "<END>",
"unk_token": "<UNK>"
}