Spaces:
Sleeping
Sleeping
| title: Token Counter | |
| emoji: 🔢 | |
| colorFrom: yellow | |
| colorTo: gray | |
| sdk: gradio | |
| sdk_version: "5.49.1" | |
| python_version: "3.12" | |
| app_file: app.py | |
| pinned: false | |
| license: mit | |
| short_description: "Count tokens across Claude, GPT, Llama tokenizers." | |
| tags: | |
| - tokenization | |
| - llm | |
| - context-window | |
| - agentfit | |
| # Token Counter | |
| Paste any text and see how it tokenizes across Claude, GPT, and other model families. Powered by [`agentfit`](https://pypi.org/project/agentfit-py/). | |
| ## Why? | |
| - Different tokenizers split the same string very differently — Claude often uses ~half the tokens GPT does for the same Chinese / emoji input. | |
| - Useful when budgeting prompts and deciding which model to use for non-English content. | |
| - Sanity-check your own token counter against a reference. | |
| ## Related | |
| - [`agentfit` on PyPI](https://pypi.org/project/agentfit-py/) | |
| - [The Agent Reliability Stack](https://mukundakatta.github.io/agent-stack/) | |
| - Companion dataset: [`token-counting-edge-cases`](https://huggingface.co/datasets/mukunda1729/token-counting-edge-cases) | |