

TITLE
GPT2Tokenizer.add_tokens() didnt change tokenizer.vocab_size

COMMENTS
1

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
follow the instruction in add_tokens()

code:

special_tokens = ['<pad>', '<go_r>', '<unk>', '<go_b>', '<go_a>', '<go_u>', ]
num_added_toks = tokenizer.add_tokens(special_tokens)
print('We have added', num_added_toks, 'tokens')

it can successfully encode and decode the newly added tokens, but the GPT2Tokenizer.vocab_size is still 50257. 