

TITLE
phase level tokenizer

COMMENTS
10

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
# 🚀 Feature request
A tokenizer to encode sentences on phase level
## Motivation
The transformer Tokenizer always tokenize the sentence on word level,which might be good for English，however it might not so in Chinese,for example, sports has single meaning in English,when its translation 运动 is split to 运 and 动,we have no idea what it means.There are also many Technical terms in chinses larger than one word should not be split

The tokenizer has additional_special_tokens, but I am not sure it can solve the phase level token
