File size: 141 Bytes
8a2b266 | 1 2 3 4 5 | from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-2")
tokenizer.save_pretrained("./ProTalk/")
|
8a2b266 | 1 2 3 4 5 | from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-2")
tokenizer.save_pretrained("./ProTalk/")
|