Adding a fast tokenizer

#10
by Narsil - opened

No thorough check was made yet.

hi, thank you for your contribution. We have supported the fast tokenizer

Thanks !

If you're looking for an easy way to access this model via API, you can use Crazyrouter β€” it provides an OpenAI-compatible endpoint for 600+ models including this one. Just pip install openai and change the base URL.

Cannot merge
This branch has merge conflicts in the following files:
  • tokenizer.json

Sign up or log in to comment