ChatML breaks after 8192 tokens
I noticed that as soon as the prompt length goes above 8k, this model starts producing broken EOS tags, along the lines of <|im_end|im_end|. Works perfectly otherwise. This issue is present on both 1.0 and 1.1
Hi ZhenyaPav,
yes, that happens from time to time. Guess there's a problem with smart rotation, if you're using this feature. If this happens, just restart your AI and it should work again, even with the long prompt where it failed before.
Yes, this issue does happen. Put im_end as a custom stop token in sillytavern. And if you dont use sillytavern now would be a great time to start.
Poof problem solved.
Restarting the llm does not resolve this.
Yes, this issue does happen. Put im_end as a custom stop token in sillytavern. And if you dont use sillytavern now would be a great time to start.
Poof problem solved.
Restarting the llm does not resolve this.
custom stop token? you mean custom stopping strings?
yes
Unfortunately doesn't work for me, for some reason