

TITLE
Difference between tokenize chinese char

COMMENTS
1

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
The function `BertTokenizer` has a parameter `tokenize_chinese_chars` and default True.
When I set it to false, I got different result as follows:
```
1. tokenize chinese char: ['任', '务']
2. not tokenize chinese char: ['任', '##务']
```
The code is as follows(and 任务 means task in English):

```
    vocab_file = './resources/robert/vocab.txt'
    bert_tokenizer1 = BertTokenizer(vocab_file, tokenize_chinese_chars=True)
    bert_tokenizer2 = BertTokenizer(vocab_file, tokenize_chinese_chars=False)
    text = '任务'
    res1 = bert_tokenizer1.tokenize(text)
    res2 = bert_tokenizer2.tokenize(text)
    print('tokenize chinese char:', res1)
    print('not tokenize chinese char:' ,res2)
```
If I use the default setting, I will get the first result. In that way, **the nearly half of vocab words will not used**(like  `'##务'`)! 
Cause we split all chinese char, it will not get `'任务'` in `WordpieceTokenizer`. It's wired.
Can somebody explain this setting for me ?