fixed tokenizer for rwkv_6_v2.1
#2
by SupYumm - opened
No description provided.
Please refer to https://github.com/jiamingkong/rwkv_tokenizer_hf/blob/main/lambada_test.md for fixed tokenizer that aligned with RWKV pip packages
Please refer to https://github.com/jiamingkong/rwkv_tokenizer_hf/blob/main/lambada_test.md for fixed tokenizer that aligned with RWKV pip packages