fixed tokenizer for rwkv_6_v2.1
#2
by
SupYumm
- opened
No description provided.
Please refer to https://github.com/jiamingkong/rwkv_tokenizer_hf/blob/main/lambada_test.md for fixed tokenizer that aligned with RWKV pip packages