| --- |
| language: |
| - kab |
| - ber |
| license: apache-2.0 |
| tags: |
| - tokenizer |
| - bpe |
| - kabyle |
| - taqbaylit |
| - tamazight |
| - nlp |
| library_name: tokenizers |
| pipeline_tag: text-generation |
| --- |
| |
| # Kabyle BPE Tokenizer v2 |
|
|
| A clean, production-ready BPE tokenizer for the Kabyle language (Taqbaylit), built from scratch with proper Unicode handling and a CLDR-grounded alphabet. |
|
|
| ## Overview |
|
|
| | Property | Value | |
| |---|---| |
| | Language | Kabyle / Taqbaylit (`kab`) | |
| | Script | Latin (Mammeri orthography) | |
| | Algorithm | Byte-Pair Encoding (BPE) | |
| | Vocab size | 48,000 | |
| | Normalizer | NFC + Lowercase | |
| | Pre-tokenizer | Metaspace + Punctuation split | |
| | Model max length | 512 | |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoTokenizer |
| |
| tok = AutoTokenizer.from_pretrained("boffire/Kabyle-BPE-Tokenizer-v2") |
| |
| text = "Taqbaylit d tutlayt tameẓẓyant yellan deg tmurt n Leqbayel." |
| ids = tok.encode(text) |
| tokens = tok.convert_ids_to_tokens(ids) |
| print(tokens) |
| # ['▁taqbaylit', '▁d', '▁tutlayt', '▁tameẓẓyant', '▁yellan', |
| # '▁deg', '▁tmurt', '▁n', '▁leqbayel', '.'] |
| ``` |
|
|
| ## Design Decisions |
|
|
| ### 1. NFC Unicode Normalization |
| Kabyle special characters (ẓ, ɣ, ɛ, ḥ…) can be encoded in two ways in UTF-8: as a single precomposed codepoint, or as a base character + combining diacritic. Without normalization, the same word can produce different token sequences depending on how the source file was saved. NFC normalization is applied before tokenization to guarantee consistency. |
|
|
| ### 2. Metaspace Pre-tokenizer |
| Instead of ByteLevel BPE (used by GPT-2-based tokenizers), this tokenizer uses a Metaspace pre-tokenizer. Word-initial spaces are marked as `▁`, and all Unicode characters — including Kabyle-specific extended Latin letters — remain as single atomic units. This eliminates the byte-level garbling (`áºĵ` for `ẓ`, `ÉĽ` for `ɛ`) seen in byte-level tokenizers. |
|
|
| ### 3. Hyphen Preserved as Morphological Marker |
| In Kabyle, the hyphen (`-`) is a morphological connector, not punctuation. It marks clitic attachment (verbal object clitics: `yewwi-t-id`), directional clitics (`yeffeɣ-d`), and possessive suffixes (`tamurt-nneɣ`). The punctuation splitter explicitly excludes `-` so these forms are pre-tokenized as single units and BPE can learn them correctly. |
|
|
| ### 4. CLDR-Grounded Alphabet |
| The `initial_alphabet` is derived exclusively from the [Unicode CLDR `kab.xml`](https://github.com/unicode-org/cldr/blob/main/common/main/kab.xml) exemplar character set: |
|
|
| **Core:** `a b c č d ḍ e ɛ f g ǧ ɣ h ḥ i j k l m n p q r ṛ s ṣ t ṭ u w x y z ẓ` |
|
|
| **Auxiliary:** `o v` |
|
|
| Every character in this set is guaranteed to exist as an atomic token in the vocabulary, regardless of its frequency in the training corpus. |
|
|
| ### 5. Distinct Special Tokens |
| Each special token has exactly one role: |
|
|
| | Token | Role | |
| |---|---| |
| | `<s>` | Beginning of sequence | |
| | `</s>` | End of sequence | |
| | `<unk>` | Unknown token | |
| | `<pad>` | Padding | |
| | `<mask>` | Masked token (for MLM) | |
|
|
| ## Benchmark |
|
|
| Comparison against 3 other publicly available Kabyle tokenizers on 10 morphologically complex Kabyle sentences (verbal clitics, negation, construct state, pronominal suffixes, dense special characters, aspectual contrast): |
|
|
| | Tokenizer | Vocab | Avg fertility ↓ | Design | |
| |---|---|---|---| |
| | `boffire/kabyle-gpt2-tokenizer` | 50,257 | 2.058 | ByteLevel BPE, broken Kabyle chars | |
| | `Hillal-titouh/kabyle-bpe-tokenizer` | 50,257 | 1.887 | No NFC, no mask token | |
| | **`boffire/kabyle-bpe-tokenizer-v2`** | **48,000** | **1.736** | ✅ NFC, CLDR alphabet, clean punctuation | |
| | `boffire/bpe-tokenizer-for-kabyle` | 48,000 | 1.476 | Absorbs punctuation into words | |
|
|
| > **Note on BPE (boffire) fertility:** Its lower fertility is structural — it absorbs punctuation marks into word tokens (e.g. `berra.` as a single token), which reduces the token count but makes punctuation boundaries invisible to the model. This tokenizer separates punctuation cleanly, which costs ~10 tokens across the benchmark but produces linguistically correct segmentation. |
|
|
| ### Sample tokenization |
|
|
| ``` |
| Yewwi-t-id ukan i gma-s. |
| → ▁yewwi- | t- | id | ▁ukan | ▁i | ▁gma- | s | . |
| |
| Ur iẓri ara acu i d-yuɣen ɣer taddart-nneɣ. |
| → ▁ur | ▁iẓri | ▁ara | ▁acu | ▁i | ▁d | - | yu | ɣen | ▁ɣer | ▁taddart- | n | neɣ | . |
| |
| Yessekcem-it deg taddart imi yeffeɣ-d ɣer berra. |
| → ▁yessekcem- | it | ▁deg | ▁taddart | ▁imi | ▁yeffeɣ- | d | ▁ɣer | ▁berra | . |
| ``` |
|
|
| ## Training Data |
|
|
| Trained on a curated Kabyle text corpus (~20 MB, ~212,000 word-level units after pre-tokenization). The corpus covers written Kabyle in the standard Mammeri Latin orthography. |
|
|
| ## Limitations |
|
|
| - **Lowercase only.** The normalizer lowercases all input. Kabyle capitalization is positional (sentence-initial) and carries no morphological information, so this is lossless for NLP tasks. Casing cannot be recovered from token IDs. |
| - **Corpus size.** At 20 MB, some low-frequency morpheme combinations (e.g. the possessive suffix `-nneɣ` after varied stems) may not have been merged and will be split across 2–3 tokens. A larger corpus will improve this. |
| - **No Tifinagh or Arabic script support.** This tokenizer covers the Latin orthography only. |
|
|
| ## Citation |
|
|
| If you use this tokenizer in your work, please cite it as: |
|
|
| ```bibtex |
| @misc{boffire2026kabyle, |
| author = {boffire}, |
| title = {Kabyle BPE Tokenizer v2}, |
| year = {2026}, |
| publisher = {HuggingFace}, |
| url = {https://huggingface.co/boffire/Kabyle-BPE-Tokenizer-v2} |
| } |
| ``` |
|
|
| ## License |
|
|
| [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0) |
