Hugging Face's logo

nanochat-students
/
nanochat-d20

PyTorch
ONNX
Safetensors
English
nanochat
Eval Results (legacy)
Model card Files
xet
 Community
13

Nanochat tokenizer conversion for HF tokenizer

#13
by Jotschi - opened
Discussion
Jotschi

@burtenshaw Can you please share how you converted the tiktokken token_bytes.pt tokenizer.pkl to the needed JSON format?
Jotschi

Found it from the Transformer Post: https://huggingface.co/spaces/nanochat-students/transformers#import-model-and-tokenizer

https://raw.githubusercontent.com/huggingface/transformers/main/src/transformers/models/nanochat/convert_nanochat_checkpoints.py

Jotschi changed discussion status to closed

Sign up or log in to comment