

TITLE
is there a tokenizer only used whitespace for spliting chinese sentence?

COMMENTS
1

REACTIONS
+1: 1
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
i want to use bert masked language model to pre train for chinese sentence,  i have alweady splited chinese sentence into meaninful word, the data file like follows:

我  是  一个  队员
他  不是  一个 合格  的 老师
......

i only want to use whitespace split them, but BertWordPieceTokenizer will split them to character level , the final vocabulary like follows:
{'[SEP]': 3,
 '一': 7,
 '是': 15,
 '员': 12,
 '[CLS]': 2,
 '[UNK]': 1,
 '</S>': 6,
 '[MASK]': 4,
 '<S>': 5,
 '队': 19,
 '的': 17,
 '不': 8,
 '我': 14,
 '他': 10,
 '老': 18,
 '[PAD]': 0,
 '格': 16,
 '个': 9,
 '师': 13,
 '合': 11}
how to correct it ?