fixed tokenizer for rwkv_6_v2.1

#2
by SupYumm - opened
No description provided.

Please refer to https://github.com/jiamingkong/rwkv_tokenizer_hf/blob/main/lambada_test.md for fixed tokenizer that aligned with RWKV pip packages

Publish this branch
This branch is in draft mode, publish it to be able to merge.

Sign up or log in to comment