Created a better tokenizer from sentencepiece model. please test it.
Thank you! I have confirmed this fixed the issue with padding.
· Sign up or log in to comment