Custom tokenizer
#1
by BuiDoan - opened
There is a mismatch in the tokenizer_config.json file. Specifically, the "tokenizer_class" property is set to "TokenizersBackend", whereas it should be "Qwen2Tokenizer" to ensure compatibility with the Hugging Face Transformers library.
raise RuntimeError(err_msg) from e
RuntimeError: Failed to load the tokenizer. If the tokenizer is a custom tokenizer not yet available in the HuggingFace transformers library, consider setting trust_remote_code=True in LLM or using the --trust-remote-code flag in the CLI.
Thanks for reporting fixing now