| --- |
| language: |
| - tr |
| - kk |
| - ky |
| - uz |
| - ug |
| - ba |
| - tt |
| - az |
| - crh |
| - tk |
| license: apache-2.0 |
| tags: |
| - tokenizer |
| - sentencepiece |
| - bpe |
| - turkic |
| - multilingual |
| library_name: transformers |
| --- |
| |
| # Pan-Turkic BPE Tokenizer |
|
|
| A SentencePiece BPE tokenizer with 65,536 vocabulary size, purpose-built for the Turkic language family. Covers Latin, Cyrillic, and Arabic scripts used across Turkic languages. |
|
|
| ## Overview |
|
|
| Most existing tokenizers fail on Turkic languages outside of Turkish — particularly on Cyrillic-script languages like Kazakh, Kyrgyz, Bashkir, and Tatar, where they fall back to byte-level tokenization. This tokenizer was trained specifically on a pan-Turkic corpus covering 20+ languages, and handles all major scripts natively. |
|
|
| **Languages with strong coverage:** |
|
|
| | Language | Script | |
| |---|---| |
| | Turkish | Latin | |
| | Kazakh | Cyrillic | |
| | Kyrgyz | Cyrillic | |
| | Uzbek | Latin | |
| | Uyghur | Arabic | |
| | Bashkir | Cyrillic | |
| | Tatar | Cyrillic | |
| | Azerbaijani | Latin | |
| | Crimean Tatar | Latin | |
| | Turkmen | Latin | |
|
|
| ## FLORES-200 Fertility Benchmark |
|
|
| Fertility = average tokens per word (lower is better). Evaluated on 1,012 sentences per language from the FLORES-200 devtest set. |
|
|
| | Language | **Ours** | Kumru-2B | GPT-2 | mT5 | NLLB-200 | XLM-R | |
| |---|---|---|---|---|---|---| |
| | Turkish | 1.78 | **1.59** | 3.79 | 2.16 | 2.00 | 1.83 | |
| | Kazakh (Cyrl) | **1.79** | 10.96 | 9.25 | 2.35 | 2.09 | 2.04 | |
| | Kyrgyz (Cyrl) | **1.73** | 11.18 | 8.95 | 2.57 | 2.21 | 2.07 | |
| | Uzbek (Latn) | **1.96** | 3.40 | 3.44 | 2.57 | 2.24 | 2.26 | |
| | Uyghur (Arab) | **1.72** | 9.42 | 10.92 | 4.91 | 2.45 | 2.46 | |
| | Bashkir (Cyrl) | **1.92** | 10.93 | 9.07 | 3.01 | 2.10 | 3.52 | |
| | Tatar (Cyrl) | **1.88** | 10.74 | 8.72 | 2.63 | 2.06 | 3.07 | |
| | Azerbaijani (Latn) | **1.72** | 3.34 | 4.92 | 2.40 | 2.16 | 1.86 | |
| | Crimean Tatar (Latn) | 2.19 | 2.61 | 3.75 | 2.49 | **2.14** | 2.36 | |
| | Turkmen (Latn) | 2.48 | 3.56 | 4.27 | 3.18 | **2.33** | 3.05 | |
| | **Turkic Avg (10 langs)** | **1.92** | 6.77 | 6.71 | 2.83 | 2.18 | 2.45 | |
| | English | 2.27 | 2.01 | **1.24** | 1.55 | 1.41 | 1.41 | |
|
|
| Vocab sizes: Ours 65,536 · Kumru-2B 50,176 · GPT-2 50,257 · mT5 250,100 · NLLB-200 256,204 · XLM-R 250,002 |
|
|
| **Key result:** Best on 7 of 10 Turkic languages. Achieves similar Turkic coverage to NLLB-200 (256K vocab) with a 4× smaller vocabulary. |
|
|
| For Cyrillic-script Turkic languages (Kazakh, Kyrgyz, Bashkir, Tatar), competing tokenizers degrade to byte-level encoding (10–11 tokens/word). This tokenizer maintains ~1.8 tokens/word on the same languages. |
|
|
| ## Notable Examples |
|
|
| Morphologically complex Turkish words encode efficiently: |
|
|
| ```python |
| "Cumhurbaşkanlığı" → 1 token # (Presidency) |
| "yapamayacaklarından" → 3 tokens # (from those they cannot do) |
| "sağlıklaştırılamayabileceklerden" → 6 tokens |
| ``` |
|
|
| Perfect round-trip for all supported scripts: |
| ```python |
| encode → decode # lossless for Latin, Cyrillic, and Arabic Turkic scripts |
| ``` |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoTokenizer |
| |
| tokenizer = AutoTokenizer.from_pretrained("ArinUmut/pan-turkic-tokenizer") |
| |
| # Turkish |
| tokenizer.encode("Türkiye Cumhuriyeti") |
| |
| # Kazakh (Cyrillic) |
| tokenizer.encode("Алматы Қазақстанның ең үлкен қаласы") |
| |
| # Uyghur (Arabic) |
| tokenizer.encode("بىز ئۇيغۇر تىلىدە سۆزلىشىمىز") |
| ``` |
|
|
| ## Specs |
|
|
| | Property | Value | |
| |---|---| |
| | Type | SentencePiece BPE | |
| | Vocabulary size | 65,536 | |
| | Scripts | Latin, Cyrillic, Arabic | |
| | Languages trained on | 20+ Turkic languages | |
| | Benchmark | FLORES-200 devtest | |
|
|
| ## Limitations |
|
|
| - English fertility (2.27) is higher than English-specialized tokenizers, as the vocabulary is optimized for Turkic languages. |
|
|
