Fixed the tokenizer configuration to get it working with Transformers v5
#2
by sorasmaa - opened
// For details about the fix, please see the issue: https://github.com/huggingface/transformers/issues/44991
// For details about the fix, please see the issue: https://github.com/huggingface/transformers/issues/44991