| --- |
| language: |
| - kab |
| - ber |
| license: mit |
| tags: |
| - tokenizer |
| - kabyle |
| - berber |
| - bpe |
| - low-resource |
| - amazigh |
| - tamaziɣt |
| - taqbaylit |
| --- |
| |
| # BPE Tokenizer for kabyle |
|
|
| A high-performance Byte-Pair Encoding tokenizer optimized for the Kabyle language (Taqbaylit). Designed for low-resource NLP and LLM training. |
|
|
| ## Key Specifications |
| - **Vocabulary Size:** 48,011 tokens |
| - **Fertility:** 1.376 tokens/word |
| - **Corpus:** ~19M characters of cleaned Kabyle text |
| - **Encoding Speed:** ~34,000 sentences/second (CPU) |
|
|
| ## Usage |
| ```python |
| from transformers import AutoTokenizer |
| |
| tokenizer = AutoTokenizer.from_pretrained("boffire/bpe-tokenizer-for-kabyle") |
| tokens = tokenizer.tokenize("aɣ-d-yini tameddurt-nneɣ deg taddart.") |
| # ['aɣ-d-yini', 'tameddurt-nneɣ', 'deg', 'taddart', '.'] |
| ``` |
|
|
| ## Performance Comparison |
| Benchmarked against the primary public Kabyle tokenizer (Hillal-titouh/kabyle-bpe-tokenizer): |
|
|
| - **Sequence Efficiency:** 15.5% lower fertility (1.376 vs 1.593 tokens/word) |
| - **Unicode Handling:** Native support for Kabyle characters (ɛ, ẓ, ṛ, ṭ, ṣ, ḍ, ǧ, ḥ, ɣ, č) without byte-level fallback |
| - **Morphological Awareness:** High-frequency prefixes, clitics, and compounds preserved as single tokens |
|
|
| ## Training Details |
| Trained using the Hugging Face tokenizers library on a deduplicated, NFC-normalized Kabyle corpus. Post-training analysis identified and corrected suboptimal BPE splits on high-frequency words by adding them to the vocabulary as fixed tokens. This ensures consistent tokenization without corrupting the BPE merge graph. |
|
|
| ## Limitations |
|
|
| - Compound token overrides are exact-match. Unseen morphological variants may still split. |
| - Optimized specifically for Kabyle. Performance may degrade on multilingual or code-switched text. |
|
|
| ## License & Citation |
|
|
| - **License:** MIT |
