Update README.md

README.md

@@ -1,3 +1,131 @@
 ---
-license: mit
+language:
+- kab
+- ber
+license: apache-2.0
+tags:
+- tokenizer
+- bpe
+- kabyle
+- taqbaylit
+- tamazight
+- nlp
+library_name: tokenizers
+pipeline_tag: text-generation
 ---
+
+# Kabyle BPE Tokenizer v2
+
+A clean, production-ready BPE tokenizer for the Kabyle language (Taqbaylit), built from scratch with proper Unicode handling and a CLDR-grounded alphabet.
+
+## Overview
+
+| Property | Value |
+|---|---|
+| Language | Kabyle / Taqbaylit (`kab`) |
+| Script | Latin (Mammeri orthography) |
+| Algorithm | Byte-Pair Encoding (BPE) |
+| Vocab size | 48,000 |
+| Normalizer | NFC + Lowercase |
+| Pre-tokenizer | Metaspace + Punctuation split |
+| Model max length | 512 |
+
+## Usage
+
+```python
+from transformers import AutoTokenizer
+
+tok = AutoTokenizer.from_pretrained("boffire/Kabyle-BPE-Tokenizer-v2")
+
+text = "Taqbaylit d tutlayt tameẓẓyant yellan deg tmurt n Leqbayel."
+ids = tok.encode(text)
+tokens = tok.convert_ids_to_tokens(ids)
+print(tokens)
+# ['▁taqbaylit', '▁d', '▁tutlayt', '▁tameẓẓyant', '▁yellan',
+#  '▁deg', '▁tmurt', '▁n', '▁leqbayel', '.']
+```
+
+## Design Decisions
+
+### 1. NFC Unicode Normalization
+Kabyle special characters (ẓ, ɣ, ɛ, ḥ…) can be encoded in two ways in UTF-8: as a single precomposed codepoint, or as a base character + combining diacritic. Without normalization, the same word can produce different token sequences depending on how the source file was saved. NFC normalization is applied before tokenization to guarantee consistency.
+
+### 2. Metaspace Pre-tokenizer
+Instead of ByteLevel BPE (used by GPT-2-based tokenizers), this tokenizer uses a Metaspace pre-tokenizer. Word-initial spaces are marked as `▁`, and all Unicode characters — including Kabyle-specific extended Latin letters — remain as single atomic units. This eliminates the byte-level garbling (`áºĵ` for `ẓ`, `ÉĽ` for `ɛ`) seen in byte-level tokenizers.
+
+### 3. Hyphen Preserved as Morphological Marker
+In Kabyle, the hyphen (`-`) is a morphological connector, not punctuation. It marks clitic attachment (verbal object clitics: `yewwi-t-id`), directional clitics (`yeffeɣ-d`), and possessive suffixes (`tamurt-nneɣ`). The punctuation splitter explicitly excludes `-` so these forms are pre-tokenized as single units and BPE can learn them correctly.
+
+### 4. CLDR-Grounded Alphabet
+The `initial_alphabet` is derived exclusively from the [Unicode CLDR `kab.xml`](https://github.com/unicode-org/cldr/blob/main/common/main/kab.xml) exemplar character set:
+
+**Core:** `a b c č d ḍ e ɛ f g ǧ ɣ h ḥ i j k l m n p q r ṛ s ṣ t ṭ u w x y z ẓ`
+
+**Auxiliary:** `o v`
+
+Every character in this set is guaranteed to exist as an atomic token in the vocabulary, regardless of its frequency in the training corpus.
+
+### 5. Distinct Special Tokens
+Each special token has exactly one role:
+
+| Token | Role |
+|---|---|
+| `<s>` | Beginning of sequence |
+| `</s>` | End of sequence |
+| `<unk>` | Unknown token |
+| `<pad>` | Padding |
+| `<mask>` | Masked token (for MLM) |
+
+## Benchmark
+
+Comparison against 3 other publicly available Kabyle tokenizers on 10 morphologically complex Kabyle sentences (verbal clitics, negation, construct state, pronominal suffixes, dense special characters, aspectual contrast):
+
+| Tokenizer | Vocab | Avg fertility ↓ | Design |
+|---|---|---|---|
+| `boffire/kabyle-gpt2-tokenizer` | 50,257 | 2.058 | ByteLevel BPE, broken Kabyle chars |
+| `Hillal-titouh/kabyle-bpe-tokenizer` | 50,257 | 1.887 | No NFC, no mask token |
+| **`boffire/kabyle-bpe-tokenizer-v2`** | **48,000** | **1.736** | ✅ NFC, CLDR alphabet, clean punctuation |
+| `boffire/bpe-tokenizer-for-kabyle` | 48,000 | 1.476 | Absorbs punctuation into words |
+
+> **Note on BPE (boffire) fertility:** Its lower fertility is structural — it absorbs punctuation marks into word tokens (e.g. `berra.` as a single token), which reduces the token count but makes punctuation boundaries invisible to the model. This tokenizer separates punctuation cleanly, which costs ~10 tokens across the benchmark but produces linguistically correct segmentation.
+
+### Sample tokenization
+
+```
+Yewwi-t-id ukan i gma-s.
+→ ▁yewwi- | t- | id | ▁ukan | ▁i | ▁gma- | s | .
+
+Ur iẓri ara acu i d-yuɣen ɣer taddart-nneɣ.
+→ ▁ur | ▁iẓri | ▁ara | ▁acu | ▁i | ▁d | - | yu | ɣen | ▁ɣer | ▁taddart- | n | neɣ | .
+
+Yessekcem-it deg taddart imi yeffeɣ-d ɣer berra.
+→ ▁yessekcem- | it | ▁deg | ▁taddart | ▁imi | ▁yeffeɣ- | d | ▁ɣer | ▁berra | .
+```
+
+## Training Data
+
+Trained on a curated Kabyle text corpus (~20 MB, ~212,000 word-level units after pre-tokenization). The corpus covers written Kabyle in the standard Mammeri Latin orthography.
+
+## Limitations
+
+- **Lowercase only.** The normalizer lowercases all input. Kabyle capitalization is positional (sentence-initial) and carries no morphological information, so this is lossless for NLP tasks. Casing cannot be recovered from token IDs.
+- **Corpus size.** At 20 MB, some low-frequency morpheme combinations (e.g. the possessive suffix `-nneɣ` after varied stems) may not have been merged and will be split across 2–3 tokens. A larger corpus will improve this.
+- **No Tifinagh or Arabic script support.** This tokenizer covers the Latin orthography only.
+
+## Citation
+
+If you use this tokenizer in your work, please cite it as:
+
+```bibtex
+@misc{boffire2024kabyle,
+  author    = {boffire},
+  title     = {Kabyle BPE Tokenizer v2},
+  year      = {2025},
+  publisher = {HuggingFace},
+  url       = {https://huggingface.co/boffire/Kabyle-BPE-Tokenizer-v2}
+}
+```
+
+## License
+
+[Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0)