Hi, I try to load with LlamaForCausalLM, LlamaTokenizer, but it show me the error that "not a string"
#64
by hjewr - opened
Hi, I try to load with LlamaForCausalLM, LlamaTokenizer, but it show me the error that "not a string".
It this because the naming problem? maybe tokenizer.json to tokenizer.model
Thank u. Yeah, it works. But I wonder what was the reason for that? the sentencepice problem?
YEs, the reason is because as we mention on the card, the tokenizer is NOT sentencepiece based
The tokenizer.model is from tiktoken. We converted it to the tokenizers format, so AutoTokenizer will use a PreTrainedTokenizerFast
Closing as solved!
ArthurZ changed discussion status to closed