

TITLE
Code bug in tokenization_utils.py?

COMMENTS
3

REACTIONS
+1: 1
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
In the function split_on_tokens in tokenization_utils.py, it contains the following logic:

if not text.strip():
    return []
if not tok_list:
    return self._tokenize(text)

So if the text contains just white space, '[]' will be returned. However, if the text contains white space and visible characters, the white space will be used in tokenization. For example, for the following code:

from transformers import GPT2Tokenizer
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
print(tokenizer("\nNorth")['input_ids'])  # output [198, 14157], since 198 <-> \n and 14157 <-> North
print(tokenizer("\n")['input_ids'])            # output [], even if 198 <-> \n

Is this behavior that we expected?