ChatML taking up many tokens
Hi, thanks for this cool fine tune, I was wondering if there is some remedy for the issue of ChatML taking up many tokens. For example, <|im_end|> is 7 tokens, which takes a lot of time to process. If it and the rest of the ChatML formatting took up only one, the speed would be increased a lot. Is there something I'm missing here or is it really supposed to be 7 tokens?
Hi, thanks for this cool fine tune, I was wondering if there is some remedy for the issue of ChatML taking up many tokens. For example, <|im_end|> is 7 tokens, which takes a lot of time to process. If it and the rest of the ChatML formatting took up only one, the speed would be increased a lot. Is there something I'm missing here or is it really supposed to be 7 tokens?
I think that is my fault. I did not add special tokens for them. For now, it seems not causing too much problem, and I will fix that in next upcoming base model update🤗
Ah, alright. Next upcoming base model update being the 1.5T?