Upload all models and assets for shi (latest)

README.md

@@ -36,738 +36,195 @@ metrics:
     value: 3.819
   - name: best_isotropy
     type: isotropy
-    value: 0.7173
+    value: 0.6948
+  - name: best_alignment_r10
+    type: alignment
+    value: 0.1780
   - name: vocabulary_size
     type: vocab
-    value: 0
-generated: 2026-01-10
+    value: 31623
+generated: 2026-03-02
 ---
 
-# Tachelhit - Wikilangs Models
-## Comprehensive Research Report & Full Ablation Study
+# Tachelhit — Wikilangs Models
 
-This repository contains NLP models trained and evaluated by Wikilangs, specifically on **Tachelhit** Wikipedia data.
-We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and word embeddings.
+Open-source tokenizers, n-gram & Markov language models, vocabulary stats, and word embeddings trained on **Tachelhit** Wikipedia by [Wikilangs](https://wikilangs.org).
 
-## 📋 Repository Contents
+🌐 [Language Page](https://wikilangs.org/languages/shi/) · 🎮 [Playground](https://wikilangs.org/playground/?lang=shi) · 📊 [Full Research Report](RESEARCH_REPORT.md)
 
-### Models & Assets
+## Language Samples
 
-- Tokenizers (8k, 16k, 32k, 64k)
-- N-gram models (2, 3, 4, 5-gram)
-- Markov chains (context of 1, 2, 3, 4 and 5)
-- Subword N-gram and Markov chains
-- Embeddings in various sizes and dimensions (aligned and unaligned)
-- Language Vocabulary
-- Language Statistics
+Example sentences drawn from the Tachelhit Wikipedia corpus:
 
-![Performance Dashboard](visualizations/performance_dashboard.png)
+> 11 yan d mraw ( s Taɛrabt احدى عشر ) ( s Tafṛensist onze ) iga yan izwl Msmun awal n SGSM : Msmun awal amatay asnmalay n tmaziɣt (MMSM) tisaɣulin
 
-### Analysis and Evaluation
+> 12 sin d mraw ( s Taɛrabt اثنى عشرة ) ( s Tafṛensist douze ) iga yan izwl Msmun awal n SGSM : Msmun awal amatay asnmalay n tmaziɣt (MMSM) tisaɣulin
 
-- [1. Tokenizer Evaluation](#1-tokenizer-evaluation)
-- [2. N-gram Model Evaluation](#2-n-gram-model-evaluation)
-- [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
-- [4. Vocabulary Analysis](#4-vocabulary-analysis)
-- [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Morphological Analysis (Experimental)](#6--morphological-analysis-experimental)
-- [7. Summary & Recommendations](#7-summary--recommendations)
-- [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
-- [Visualizations Index](#visualizations-index)
+> 13 kṛaḍ d merraw ( s Taɛrabt ثلاثة عشرة ) ( s Tafṛensist treize ) iga yan izwl Msmun awal n SGSM : Msmun awal amatay asnmalay n tmaziɣt (MMSM) tisaɣulin
 
----
-## 1. Tokenizer Evaluation
+> Acfud iga yat tasklut mẓẓin, ilan isnnann. Tiwlafin Assaɣ Tasnalɣa (morphologie) Anzwi Tisaɣulin
 
-![Tokenizer Compression](visualizations/tokenizer_compression.png)
+> Acnyal nɣ Aknyal agdudan aṣbnyuli, ɣ tgzzumt tiss 4.1 n tmnḍawt taṣbnyulit yuma kṛaḍ ikʷlan: aẓggaɣ d uwraɣ d uẓggaɣ daɣ. Tisaɣulin
 
-![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+## Quick Start
 
-![Tokenizer OOV](visualizations/tokenizer_oov.png)
+### Load the Tokenizer
 
-![Total Tokens](visualizations/tokenizer_total_tokens.png)
+```python
+import sentencepiece as spm
 
-### Results
+sp = spm.SentencePieceProcessor()
+sp.Load("shi_tokenizer_32k.model")
 
-| Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
-|------------|-------------|---------------|----------|--------------|
-| **8k** | 3.016x | 3.02 | 1.3945% | 407,897 |
-| **16k** | 3.301x | 3.30 | 1.5260% | 372,731 |
-| **32k** | 3.556x | 3.56 | 1.6440% | 345,980 |
-| **64k** | 3.819x 🏆 | 3.82 | 1.7653% | 322,212 |
+text = "Sstekk iga yan ugḍiḍ imẓẓin. Assaɣ Tuzduɣt Tasnalɣa (morphologie) Tisaɣulin Msmu"
+tokens = sp.EncodeAsPieces(text)
+ids    = sp.EncodeAsIds(text)
 
-### Tokenization Examples
+print(tokens)  # subword pieces
+print(ids)     # integer ids
 
-Below are sample sentences tokenized with each vocabulary size:
+# Decode back
+print(sp.DecodeIds(ids))
+```
 
-**Sample 1:** `Talggut neɣ Algu tga yat tasklut ur iskaren awd yat ugummu, tesker ifrawen zund ...`
+<details>
+<summary><b>Tokenization examples (click to expand)</b></summary>
+
+**Sample 1:** `Sstekk iga yan ugḍiḍ imẓẓin. Assaɣ Tuzduɣt Tasnalɣa (morphologie) Tisaɣulin Msmu…`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁tal gg ut ▁neɣ ▁al gu ▁tga ▁yat ▁tas klut ... (+29 more)` | 39 |
-| 16k | `▁tal ggut ▁neɣ ▁algu ▁tga ▁yat ▁tasklut ▁ur ▁iskar en ... (+22 more)` | 32 |
-| 32k | `▁talggut ▁neɣ ▁algu ▁tga ▁yat ▁tasklut ▁ur ▁iskaren ▁awd ▁yat ... (+20 more)` | 30 |
-| 64k | `▁talggut ▁neɣ ▁algu ▁tga ▁yat ▁tasklut ▁ur ▁iskaren ▁awd ▁yat ... (+19 more)` | 29 |
+| 8k | `▁s ste kk ▁iga ▁yan ▁ugḍiḍ ▁imẓẓin . ▁assaɣ ▁tuzduɣt … (+19 more)` | 29 |
+| 16k | `▁s ste kk ▁iga ▁yan ▁ugḍiḍ ▁imẓẓin . ▁assaɣ ▁tuzduɣt … (+19 more)` | 29 |
+| 32k | `▁s stekk ▁iga ▁yan ▁ugḍiḍ ▁imẓẓin . ▁assaɣ ▁tuzduɣt ▁tasnal��a … (+18 more)` | 28 |
+| 64k | `▁sstekk ▁iga ▁yan ▁ugḍiḍ ▁imẓẓin . ▁assaɣ ▁tuzduɣt ▁tasnalɣa ▁( … (+17 more)` | 27 |
 
-**Sample 2:** `1 000 iga yan umḍan imqquṛn, ism ns s tmaziɣt igat ifḍ (s tfinaɣ : ⵉⴼⴹ). Msmun a...`
+**Sample 2:** `Asimwas iga ass wiss Smmus g ussan n imalass. Tisaɣulin`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁ 1 ▁ 0 0 0 ▁iga ▁yan ▁umḍan ▁imqquṛn ... (+30 more)` | 40 |
-| 16k | `▁ 1 ▁ 0 0 0 ▁iga ▁yan ▁umḍan ▁imqquṛn ... (+29 more)` | 39 |
-| 32k | `▁ 1 ▁ 0 0 0 ▁iga ▁yan ▁umḍan ▁imqquṛn ... (+29 more)` | 39 |
-| 64k | `▁ 1 ▁ 0 0 0 ▁iga ▁yan ▁umḍan ▁imqquṛn ... (+29 more)` | 39 |
+| 8k | `▁as im was ▁iga ▁ass ▁wiss ▁smmus ▁g ▁ussan ▁n … (+3 more)` | 13 |
+| 16k | `▁as imwas ▁iga ▁ass ▁wiss ▁smmus ▁g ▁ussan ▁n ▁imalass … (+2 more)` | 12 |
+| 32k | `▁asimwas ▁iga ▁ass ▁wiss ▁smmus ▁g ▁ussan ▁n ▁imalass . … (+1 more)` | 11 |
+| 64k | `▁asimwas ▁iga ▁ass ▁wiss ▁smmus ▁g ▁ussan ▁n ▁imalass . … (+1 more)` | 11 |
 
-**Sample 3:** `Iga Q yan sg iskkiln n ugmmay alatin n tmaziɣt. Tisaɣulin amaziɣ tamaziɣt`
+**Sample 3:** `Turdut (S turdut: اردو ) tga tutlayt nna s sawaln ayt Bakistan d Lhnd. Isuɣal`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁iga ▁q ▁yan ▁sg ▁iskkiln ▁n ▁ugmmay ▁alatin ▁n ▁tmaziɣt ... (+4 more)` | 14 |
-| 16k | `▁iga ▁q ▁yan ▁sg ▁iskkiln ▁n ▁ugmmay ▁alatin ▁n ▁tmaziɣt ... (+4 more)` | 14 |
-| 32k | `▁iga ▁q ▁yan ▁sg ▁iskkiln ▁n ▁ugmmay ▁alatin ▁n ▁tmaziɣt ... (+4 more)` | 14 |
-| 64k | `▁iga ▁q ▁yan ▁sg ▁iskkiln ▁n ▁ugmmay ▁alatin ▁n ▁tmaziɣt ... (+4 more)` | 14 |
-
-
-### Key Findings
-
-- **Best Compression:** 64k achieves 3.819x compression
-- **Lowest UNK Rate:** 8k with 1.3945% unknown tokens
-- **Trade-off:** Larger vocabularies improve compression but increase model size
-- **Recommendation:** 32k vocabulary provides optimal balance for production use
-
----
-## 2. N-gram Model Evaluation
-
-![N-gram Perplexity](visualizations/ngram_perplexity.png)
-
-![N-gram Unique](visualizations/ngram_unique.png)
-
-![N-gram Coverage](visualizations/ngram_coverage.png)
-
-### Results
-
-| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
-|--------|---------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | Word | 1,027 | 10.00 | 23,236 | 45.7% | 81.7% |
-| **2-gram** | Subword | 255 🏆 | 7.99 | 3,781 | 68.8% | 99.0% |
-| **3-gram** | Word | 1,698 | 10.73 | 46,052 | 39.0% | 76.4% |
-| **3-gram** | Subword | 1,284 | 10.33 | 29,091 | 35.1% | 84.7% |
-| **4-gram** | Word | 3,109 | 11.60 | 90,307 | 35.2% | 68.9% |
-| **4-gram** | Subword | 3,344 | 11.71 | 117,787 | 23.5% | 73.6% |
-| **5-gram** | Word | 3,900 | 11.93 | 100,603 | 35.2% | 65.7% |
-| **5-gram** | Subword | 5,685 | 12.47 | 238,802 | 18.6% | 68.5% |
-
-### Top 5 N-grams by Size
-
-**2-grams (Word):**
-
-| Rank | N-gram | Count |
-|------|--------|-------|
-| 1 | `tgmiḍi n` | 30,047 |
-| 2 | `n usggʷas` | 27,406 |
-| 3 | `umḍan n` | 26,921 |
-| 4 | `n imzdaɣn` | 25,250 |
-| 5 | `tlkm tgmiḍi` | 24,096 |
-
-**3-grams (Word):**
-
-| Rank | N-gram | Count |
-|------|--------|-------|
-| 1 | `tlkm tgmiḍi n` | 24,096 |
-| 2 | `tamattayt n usɣiws` | 16,122 |
-| 3 | `tasmirit tamattayt n` | 15,740 |
-| 4 | `umḍan n imzdaɣn` | 14,946 |
-| 5 | `g tlkm tgmiḍi` | 12,050 |
-
-**4-grams (Word):**
-
-| Rank | N-gram | Count |
-|------|--------|-------|
-| 1 | `tasmirit tamattayt n usɣiws` | 15,739 |
-| 2 | `g tlkm tgmiḍi n` | 12,050 |
-| 3 | `ad i trfiqt n` | 8,924 |
-| 4 | `uḍwwaṛ ad i trfiqt` | 8,917 |
-| 5 | `umḍan n imzdaɣn nns` | 8,916 |
-
-**5-grams (Word):**
-
-| Rank | N-gram | Count |
-|------|--------|-------|
-| 1 | `uḍwwaṛ ad i trfiqt n` | 8,916 |
-| 2 | `amatay n imzdaɣn tasmirit tamattayt` | 8,910 |
-| 3 | `imzdaɣn tasmirit tamattayt n usɣiws` | 8,910 |
-| 4 | `n imzdaɣn tasmirit tamattayt n` | 8,910 |
-| 5 | `ilkm umḍan n imzdaɣn nns` | 8,904 |
-
-**2-grams (Subword):**
-
-| Rank | N-gram | Count |
-|------|--------|-------|
-| 1 | `n _` | 653,867 |
-| 2 | `_ n` | 401,914 |
-| 3 | `_ t` | 358,373 |
-| 4 | `_ i` | 253,323 |
-| 5 | `t a` | 205,156 |
-
-**3-grams (Subword):**
-
-| Rank | N-gram | Count |
-|------|--------|-------|
-| 1 | `_ n _` | 294,487 |
-| 2 | `_ t a` | 132,536 |
-| 3 | `n _ t` | 104,627 |
-| 4 | `a n _` | 103,501 |
-| 5 | `_ ɣ _` | 101,865 |
-
-**4-grams (Subword):**
-
-| Rank | N-gram | Count |
-|------|--------|-------|
-| 1 | `_ n _ u` | 84,430 |
-| 2 | `t _ n _` | 67,376 |
-| 3 | `_ n _ i` | 61,495 |
-| 4 | `_ n _ t` | 56,122 |
-| 5 | `n _ u s` | 52,239 |
-
-**5-grams (Subword):**
-
-| Rank | N-gram | Count |
-|------|--------|-------|
-| 1 | `_ n _ u s` | 51,413 |
-| 2 | `m z d a ɣ` | 46,710 |
-| 3 | `g g ʷ a s` | 34,963 |
-| 4 | `s g g ʷ a` | 34,938 |
-| 5 | `_ n n a _` | 34,315 |
-
-
-### Key Findings
-
-- **Best Perplexity:** 2-gram (subword) with 255
-- **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~68% of corpus
-- **Recommendation:** 4-gram or 5-gram for best predictive performance
-
----
-## 3. Markov Chain Evaluation
-
-![Markov Entropy](visualizations/markov_entropy.png)
-
-![Markov Contexts](visualizations/markov_contexts.png)
-
-![Markov Branching](visualizations/markov_branching.png)
-
-### Results
-
-| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
-|---------|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | Word | 0.6330 | 1.551 | 4.06 | 76,235 | 36.7% |
-| **1** | Subword | 1.2937 | 2.452 | 10.38 | 803 | 0.0% |
-| **2** | Word | 0.2598 | 1.197 | 1.65 | 308,778 | 74.0% |
-| **2** | Subword | 1.0718 | 2.102 | 6.52 | 8,338 | 0.0% |
-| **3** | Word | 0.0839 | 1.060 | 1.19 | 508,428 | 91.6% |
-| **3** | Subword | 0.8300 | 1.778 | 3.82 | 54,347 | 17.0% |
-| **4** | Word | 0.0475 🏆 | 1.033 | 1.13 | 601,160 | 95.2% |
-| **4** | Subword | 0.5641 | 1.478 | 2.43 | 207,735 | 43.6% |
-
-### Generated Text Samples (Word-based)
-
-Below are text samples generated from each word-based Markov chain model:
-
-**Context Size 1:**
-
-1. `n ayt twayya nna dar irgazn d amatay n usggʷas niɣ uggar ɣ ɛin ijri s`
-2. `ɣ uḍwwaṛ innazwan yili ɣ lmɣrib iḍfaṛ uḍwwaṛ ad i twuri tannayin tisaɣulin ɣ llan 4`
-3. `d 11 n tarwuri 2 ig unammas n tznit tamnaḍt n iḍuṛan ilkm wawtay nnsn iẓḍiṛn`
-
-**Context Size 2:**
-
-1. `tgmiḍi n uslmd 92 86 gr irban d trbatin nna dar 15 n usggʷas démographiques et socio`
-2. `n usggʷas démographiques et socio économiques de la population rurale hors nomades par douar selon l...`
-3. `umḍan n imzdaɣn n usun ad 20 n iḍuṛan ilkm umḍan n twjiwin s 32 7 gr`
-
-**Context Size 3:**
-
-1. `tlkm tgmiḍi n uslmd 100 gr irban d trbatin nna dar gr 6 d 11 n usggʷas ɣ`
-2. `tamattayt n usɣiws tannayin tisaɣulin ɣ lmɣrib ɣ tsga n lḥuz n lḥuz n lḥuz n lḥuz n`
-3. `tasmirit tamattayt n usɣiws ɣ iga umḍan n imawaḍn 224 n umzdaɣ gisn 581 n iwtman d 329`
-
-**Context Size 4:**
-
-1. `tasmirit tamattayt n usɣiws ɣ iga umḍan n imawaḍn 236 n umzdaɣ gisn 110 n iwtman d 101 n`
-2. `g tlkm tgmiḍi n uslmd 89 66 gr irban d trbatin nna dar gr 6 d 11 n usggʷas`
-3. `ad i trfiqt n ayt iɛzman nna ɣ llan 4 n iḍuṛan ilkm umḍan n imzdaɣn nns 251 n`
-
-
-### Generated Text Samples (Subword-based)
-
-Below are text samples generated from each subword-based Markov chain model:
-
-**Context Size 1:**
-
-1. `_5_wtaṛsɣ_t_tana`
-2. `aḍas_tm_aɣnaphon`
-3. `nn_nn_puriquriɣ_`
-
-**Context Size 2:**
-
-1. `n_muḍwwawtmas_soc`
-2. `_n_et_tamklattamk`
-3. `_tawtmadin_tlkm_u`
-
-**Context Size 3:**
+| 8k | `▁tur dut ▁( s ▁tur dut : ▁ا ر دو … (+14 more)` | 24 |
+| 16k | `▁tur dut ▁( s ▁tur dut : ▁ار دو ▁) … (+13 more)` | 23 |
+| 32k | `▁turdut ▁( s ▁turdut : ▁اردو ▁) ▁tga ▁tutlayt ▁nna … (+9 more)` | 19 |
+| 64k | `▁turdut ▁( s ▁turdut : ▁اردو ▁) ▁tga ▁tutlayt ▁nna … (+8 more)` | 18 |
 
-1. `_n_imzdaɣn_nit_soc`
-2. `_tarwurin_i_trfiqt`
-3. `n_tawuri._tluḥarch`
+</details>
 
-**Context Size 4:**
+### Load Word Embeddings
 
-1. `_n_usɣiws._aṛcif,_1`
-2. `t_n_iwtman_d_23.95_`
-3. `_n_imzdaɣn_n_iwtman`
+```python
+from gensim.models import KeyedVectors
 
+# Aligned embeddings (cross-lingual, mapped to English vector space)
+wv = KeyedVectors.load("shi_embeddings_128d_aligned.kv")
 
-### Key Findings
-
-- **Best Predictability:** Context-4 (word) with 95.2% predictability
-- **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (207,735 contexts)
-- **Recommendation:** Context-3 or Context-4 for text generation
-
----
-## 4. Vocabulary Analysis
-
-![Zipf's Law](visualizations/zipf_law.png)
-
-![Top Words](visualizations/top20_words.png)
-
-![Coverage Curve](visualizations/vocab_coverage.png)
-
-### Statistics
-
-| Metric | Value |
-|--------|-------|
-| Vocabulary Size | 31,610 |
-| Total Tokens | 2,378,642 |
-| Mean Frequency | 75.25 |
-| Median Frequency | 4 |
-| Frequency Std Dev | 1969.69 |
-
-### Most Common Words
-
-| Rank | Word | Frequency |
-|------|------|-----------|
-| 1 | n | 294,685 |
-| 2 | ɣ | 101,988 |
-| 3 | d | 64,374 |
-| 4 | s | 34,997 |
-| 5 | nna | 34,361 |
-| 6 | imzdaɣn | 31,398 |
-| 7 | dar | 30,865 |
-| 8 | gr | 30,721 |
-| 9 | tgmiḍi | 30,050 |
-| 10 | usggʷas | 28,210 |
-
-### Least Common Words (from vocabulary)
-
-| Rank | Word | Frequency |
-|------|------|-----------|
-| 1 | tdarwinit | 2 |
-| 2 | talmuqqdimt | 2 |
-| 3 | ttawnn | 2 |
-| 4 | taggrgist | 2 |
-| 5 | umdgar | 2 |
-| 6 | uqṛiḍ | 2 |
-| 7 | dearborn | 2 |
-| 8 | ghosts | 2 |
-| 9 | tremblay | 2 |
-| 10 | tmmndl | 2 |
-
-### Zipf's Law Analysis
-
-| Metric | Value |
-|--------|-------|
-| Zipf Coefficient | 1.2849 |
-| R² (Goodness of Fit) | 0.988016 |
-| Adherence Quality | **excellent** |
-
-### Coverage Analysis
-
-| Top N Words | Coverage |
-|-------------|----------|
-| Top 100 | 69.6% |
-| Top 1,000 | 90.6% |
-| Top 5,000 | 95.6% |
-| Top 10,000 | 97.3% |
-
-### Key Findings
-
-- **Zipf Compliance:** R²=0.9880 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 69.6% of corpus
-- **Long Tail:** 21,610 words needed for remaining 2.7% coverage
-
----
-## 5. Word Embeddings Evaluation
-
-![Embedding Isotropy](visualizations/embedding_isotropy.png)
-
-![Similarity Matrix](visualizations/embedding_similarity.png)
-
-![t-SNE Words](visualizations/tsne_words.png)
-
-![t-SNE Sentences](visualizations/tsne_sentences.png)
-
-
-### 5.1 Cross-Lingual Alignment
-
-![Alignment Quality](visualizations/embedding_alignment_quality.png)
-
-![Multilingual t-SNE](visualizations/embedding_tsne_multilingual.png)
-
-
-### 5.2 Model Comparison
-
-| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
-|-------|-----------|----------|------------------|---------------|----------------|
-| **mono_32d** | 32 | 0.7173 | 0.3723 | N/A | N/A |
-| **mono_64d** | 64 | 0.5707 | 0.3238 | N/A | N/A |
-| **mono_128d** | 128 | 0.2225 | 0.3121 | N/A | N/A |
-| **aligned_32d** | 32 | 0.7173 🏆 | 0.3624 | 0.0140 | 0.0980 |
-| **aligned_64d** | 64 | 0.5707 | 0.3343 | 0.0280 | 0.1200 |
-| **aligned_128d** | 128 | 0.2225 | 0.3186 | 0.0400 | 0.1960 |
+similar = wv.most_similar("word", topn=5)
+for word, score in similar:
+    print(f"  {word}: {score:.3f}")
+```
 
-### Key Findings
+### Load N-gram Model
 
-- **Best Isotropy:** aligned_32d with 0.7173 (more uniform distribution)
-- **Semantic Density:** Average pairwise similarity of 0.3372. Lower values indicate better semantic separation.
-- **Alignment Quality:** Aligned models achieve up to 4.0% R@1 in cross-lingual retrieval.
-- **Recommendation:** 128d aligned for best cross-lingual performance
+```python
+import pyarrow.parquet as pq
 
----
-## 6.  Morphological Analysis (Experimental)
-
-This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
-
-### 6.1 Productivity & Complexity
-
-| Metric | Value | Interpretation | Recommendation |
-|--------|-------|----------------|----------------|
-| Productivity Index | **5.000** | High morphological productivity | Reliable analysis |
-| Idiomaticity Gap | **-0.041** | Low formulaic content | - |
-
-### 6.2 Affix Inventory (Productive Units)
-
-These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
-
-#### Productive Prefixes
-| Prefix | Examples |
-|--------|----------|
-| `-t` | tugt, tattuyt, tyyuga |
-| `-i` | issiks, imẓyann, izdg |
-| `-ta` | tattuyt, taryal, tamaẓuẓt |
-| `-a` | azdawan, amazɣ, afnsu |
-| `-u` | utin, uswaɣ, uzzugz |
-| `-l` | lmṣalḥa, lbkr, lmujawharat |
-| `-ti` | tizrigin, tidzi, timdst |
-| `-m` | maskurt, mḥda, mmaggarn |
-
-#### Productive Suffixes
-| Suffix | Examples |
-|--------|----------|
-| `-n` | utin, azdawan, tɣmriwin |
-| `-t` | tugt, tattuyt, trifiyt |
-| `-a` | tyyuga, mḥda, tssa |
-| `-in` | utin, tɣmriwin, ɣwin |
-| `-s` | issiks, chaouis, nations |
-| `-i` | inlbi, uɣri, igiddi |
-| `-e` | conduite, historique, déchirée |
-| `-an` | azdawan, zyyan, franslyan |
-
-### 6.3 Bound Stems (Lexical Roots)
-
-Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
-
-| Stem | Cohesion | Substitutability | Examples |
-|------|----------|------------------|----------|
-| `adda` | 1.65x | 52 contexts | addad, wadda, jadda |
-| `ggʷa` | 1.63x | 43 contexts | aggʷa, ḥggʷa, zggʷar |
-| `ggar` | 1.94x | 22 contexts | iggar, uggar, ggarn |
-| `ugga` | 1.94x | 21 contexts | uggar, uggan, yugga |
-| `wuri` | 1.68x | 30 contexts | twuri, iswuri, swurin |
-| `tion` | 2.09x | 14 contexts | notion, action, nation |
-| `ɣrib` | 1.80x | 20 contexts | aɣrib, mɣrib, lɣribi |
-| `lati` | 1.61x | 27 contexts | latin, latif, mulati |
-| `matt` | 1.60x | 26 contexts | matta, tmatti, umatta |
-| `mɣri` | 1.79x | 13 contexts | tmɣri, mɣrib, imɣri |
-| `atio` | 1.86x | 8 contexts | nation, nations, national |
-| `mata` | 1.45x | 14 contexts | amata, smata, umata |
-
-### 6.4 Affix Compatibility (Co-occurrence)
-
-This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
-
-| Prefix | Suffix | Frequency | Examples |
-|--------|--------|-----------|----------|
-| `-t` | `-t` | 610 words | tifrirt, tdrfit |
-| `-i` | `-n` | 465 words | ittmttatn, ibṛbbachn |
-| `-t` | `-n` | 321 words | ttyussanin, tigtfulin |
-| `-t` | `-in` | 263 words | ttyussanin, tigtfulin |
-| `-l` | `-a` | 84 words | lbṛaṭla, lɛnabsa |
-| `-t` | `-a` | 65 words | tiṛṛuyṣa, tzuna |
-| `-i` | `-an` | 45 words | inultan, ilawan |
-| `-a` | `-i` | 39 words | adarazi, abriṭani |
-| `-a` | `-n` | 38 words | agwensan, agaman |
-| `-l` | `-t` | 32 words | lfwarat, lfuqqiyyat |
-
-### 6.5 Recursive Morpheme Segmentation
-
-Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
-
-| Word | Suggested Split | Confidence | Stem |
-|------|-----------------|------------|------|
-| tasnmḍant | **`tasnmḍ-an-t`** | 7.5 | `an` |
-| africaine | **`africa-in-e`** | 7.5 | `in` |
-| ttyawssannin | **`ttyawssan-n-in`** | 7.5 | `n` |
-| ittyurnan | **`ittyur-n-an`** | 7.5 | `n` |
-| ittusɣẓnn | **`ittusɣẓ-n-n`** | 7.5 | `n` |
-| zzuzzarnit | **`zzuzzar-n-it`** | 7.5 | `n` |
-| tutlayyin | **`tutlay-y-in`** | 7.5 | `y` |
-| ttaggʷanin | **`ttaggʷa-n-in`** | 7.5 | `n` |
-| marocaines | **`maroca-in-es`** | 7.5 | `in` |
-| government | **`governme-n-t`** | 7.5 | `n` |
-| ttussiḍannt | **`ttussiḍ-an-nt`** | 7.5 | `an` |
-| ittyawstay | **`ittyaws-t-ay`** | 7.5 | `t` |
-| patrimoine | **`patrimo-in-e`** | 7.5 | `in` |
-| ittuzdaɣn | **`it-tu-zdaɣn`** | 6.0 | `zdaɣn` |
-| tinsmunin | **`ti-nsmun-in`** | 6.0 | `nsmun` |
-
-### 6.6 Linguistic Interpretation
-
-> **Automated Insight:**
-The language Tachelhit shows high morphological productivity. The subword models are significantly more efficient than word models, suggesting a rich system of affixation or compounding.
+df = pq.read_table("shi_3gram_word.parquet").to_pandas()
+print(df.head())
+```
 
----
-## 7. Summary & Recommendations
+## Models Overview
 
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
-### Production Recommendations
-
-| Component | Recommended | Rationale |
-|-----------|-------------|-----------|
-| Tokenizer | **64k BPE** | Best compression (3.82x) |
-| N-gram | **2-gram** | Lowest perplexity (255) |
-| Markov | **Context-4** | Highest predictability (95.2%) |
-| Embeddings | **100d** | Balanced semantic capture and isotropy |
-
-
----
-## Appendix: Metrics Glossary & Interpretation Guide
-
-This section provides definitions, intuitions, and guidance for interpreting the metrics used throughout this report.
-
-### Tokenizer Metrics
-
-**Compression Ratio**
-> *Definition:* The ratio of characters to tokens (chars/token). Measures how efficiently the tokenizer represents text.
->
-> *Intuition:* Higher compression means fewer tokens needed to represent the same text, reducing sequence lengths for downstream models. A 3x compression means ~3 characters per token on average.
->
-> *What to seek:* Higher is generally better for efficiency, but extremely high compression may indicate overly aggressive merging that loses morphological information.
-
-**Average Token Length (Fertility)**
-> *Definition:* Mean number of characters per token produced by the tokenizer.
->
-> *Intuition:* Reflects the granularity of tokenization. Longer tokens capture more context but may struggle with rare words; shorter tokens are more flexible but increase sequence length.
->
-> *What to seek:* Balance between 2-5 characters for most languages. Arabic/morphologically-rich languages may benefit from slightly longer tokens.
-
-**Unknown Token Rate (OOV Rate)**
-> *Definition:* Percentage of tokens that map to the unknown/UNK token, indicating words the tokenizer cannot represent.
->
-> *Intuition:* Lower OOV means better vocabulary coverage. High OOV indicates the tokenizer encounters many unseen character sequences.
->
-> *What to seek:* Below 1% is excellent; below 5% is acceptable. BPE tokenizers typically achieve very low OOV due to subword fallback.
-
-### N-gram Model Metrics
-
-**Perplexity**
-> *Definition:* Measures how "surprised" the model is by test data. Mathematically: 2^(cross-entropy). Lower values indicate better prediction.
->
-> *Intuition:* If perplexity is 100, the model is as uncertain as if choosing uniformly among 100 options at each step. A perplexity of 10 means effectively choosing among 10 equally likely options.
->
-> *What to seek:* Lower is better. Perplexity decreases with larger n-grams (more context). Values vary widely by language and corpus size.
-
-**Entropy**
-> *Definition:* Average information content (in bits) needed to encode the next token given the context. Related to perplexity: perplexity = 2^entropy.
->
-> *Intuition:* High entropy means high uncertainty/randomness; low entropy means predictable patterns. Natural language typically has entropy between 1-4 bits per character.
->
-> *What to seek:* Lower entropy indicates more predictable text patterns. Entropy should decrease as n-gram size increases.
-
-**Coverage (Top-K)**
-> *Definition:* Percentage of corpus occurrences explained by the top K most frequent n-grams.
->
-> *Intuition:* High coverage with few patterns indicates repetitive/formulaic text; low coverage suggests diverse vocabulary usage.
->
-> *What to seek:* Depends on use case. For language modeling, moderate coverage (40-60% with top-1000) is typical for natural text.
-
-### Markov Chain Metrics
-
-**Average Entropy**
-> *Definition:* Mean entropy across all contexts, measuring average uncertainty in next-word prediction.
->
-> *Intuition:* Lower entropy means the model is more confident about what comes next. Context-1 has high entropy (many possible next words); Context-4 has low entropy (few likely continuations).
->
-> *What to seek:* Decreasing entropy with larger context sizes. Very low entropy (<0.1) indicates highly deterministic transitions.
-
-**Branching Factor**
-> *Definition:* Average number of unique next tokens observed for each context.
->
-> *Intuition:* High branching = many possible continuations (flexible but uncertain); low branching = few options (predictable but potentially repetitive).
->
-> *What to seek:* Branching factor should decrease with context size. Values near 1.0 indicate nearly deterministic chains.
-
-**Predictability**
-> *Definition:* Derived metric: (1 - normalized_entropy) × 100%. Indicates how deterministic the model's predictions are.
->
-> *Intuition:* 100% predictability means the next word is always certain; 0% means completely random. Real text falls between these extremes.
->
-> *What to seek:* Higher predictability for text generation quality, but too high (>98%) may produce repetitive output.
-
-### Vocabulary & Zipf's Law Metrics
-
-**Zipf's Coefficient**
-> *Definition:* The slope of the log-log plot of word frequency vs. rank. Zipf's law predicts this should be approximately -1.
->
-> *Intuition:* A coefficient near -1 indicates the corpus follows natural language patterns where a few words are very common and most words are rare.
->
-> *What to seek:* Values between -0.8 and -1.2 indicate healthy natural language distribution. Deviations may suggest domain-specific or artificial text.
-
-**R² (Coefficient of Determination)**
-> *Definition:* Measures how well the linear fit explains the frequency-rank relationship. Ranges from 0 to 1.
->
-> *Intuition:* R² near 1.0 means the data closely follows Zipf's law; lower values indicate deviation from expected word frequency patterns.
->
-> *What to seek:* R² > 0.95 is excellent; > 0.99 indicates near-perfect Zipf adherence typical of large natural corpora.
-
-**Vocabulary Coverage**
-> *Definition:* Cumulative percentage of corpus tokens accounted for by the top N words.
->
-> *Intuition:* Shows how concentrated word usage is. If top-100 words cover 50% of text, the corpus relies heavily on common words.
->
-> *What to seek:* Top-100 covering 30-50% is typical. Higher coverage indicates more repetitive text; lower suggests richer vocabulary.
-
-### Word Embedding Metrics
-
-**Isotropy**
-> *Definition:* Measures how uniformly distributed vectors are in the embedding space. Computed as the ratio of minimum to maximum singular values.
->
-> *Intuition:* High isotropy (near 1.0) means vectors spread evenly in all directions; low isotropy means vectors cluster in certain directions, reducing expressiveness.
->
-> *What to seek:* Higher isotropy generally indicates better-quality embeddings. Values > 0.1 are reasonable; > 0.3 is good. Lower-dimensional embeddings tend to have higher isotropy.
-
-**Average Norm**
-> *Definition:* Mean magnitude (L2 norm) of word vectors in the embedding space.
->
-> *Intuition:* Indicates the typical "length" of vectors. Consistent norms suggest stable training; high variance may indicate some words are undertrained.
->
-> *What to seek:* Relatively consistent norms across models. The absolute value matters less than consistency (low std deviation).
-
-**Cosine Similarity**
-> *Definition:* Measures angular similarity between vectors, ranging from -1 (opposite) to 1 (identical direction).
->
-> *Intuition:* Words with similar meanings should have high cosine similarity. This is the standard metric for semantic relatedness in embeddings.
->
-> *What to seek:* Semantically related words should score > 0.5; unrelated words should be near 0. Synonyms often score > 0.7.
-
-**t-SNE Visualization**
-> *Definition:* t-Distributed Stochastic Neighbor Embedding - a dimensionality reduction technique that preserves local structure for visualization.
->
-> *Intuition:* Clusters in t-SNE plots indicate groups of semantically related words. Spread indicates vocabulary diversity; tight clusters suggest semantic coherence.
->
-> *What to seek:* Meaningful clusters (e.g., numbers together, verbs together). Avoid over-interpreting distances - t-SNE preserves local, not global, structure.
-
-### General Interpretation Guidelines
-
-1. **Compare within model families:** Metrics are most meaningful when comparing models of the same type (e.g., 8k vs 64k tokenizer).
-2. **Consider trade-offs:** Better performance on one metric often comes at the cost of another (e.g., compression vs. OOV rate).
-3. **Context matters:** Optimal values depend on downstream tasks. Text generation may prioritize different metrics than classification.
-4. **Corpus influence:** All metrics are influenced by corpus characteristics. Wikipedia text differs from social media or literature.
-5. **Language-specific patterns:** Morphologically rich languages (like Arabic) may show different optimal ranges than analytic languages.
-
-
-### Visualizations Index
-
-| Visualization | Description |
-|---------------|-------------|
-| Tokenizer Compression | Compression ratios by vocabulary size |
-| Tokenizer Fertility | Average token length by vocabulary |
-| Tokenizer OOV | Unknown token rates |
-| Tokenizer Total Tokens | Total tokens by vocabulary |
-| N-gram Perplexity | Perplexity by n-gram size |
-| N-gram Entropy | Entropy by n-gram size |
-| N-gram Coverage | Top pattern coverage |
-| N-gram Unique | Unique n-gram counts |
-| Markov Entropy | Entropy by context size |
-| Markov Branching | Branching factor by context |
-| Markov Contexts | Unique context counts |
-| Zipf's Law | Frequency-rank distribution with fit |
-| Vocab Frequency | Word frequency distribution |
-| Top 20 Words | Most frequent words |
-| Vocab Coverage | Cumulative coverage curve |
-| Embedding Isotropy | Vector space uniformity |
-| Embedding Norms | Vector magnitude distribution |
-| Embedding Similarity | Word similarity heatmap |
-| Nearest Neighbors | Similar words for key terms |
-| t-SNE Words | 2D word embedding visualization |
-| t-SNE Sentences | 2D sentence embedding visualization |
-| Position Encoding | Encoding method comparison |
-| Model Sizes | Storage requirements |
-| Performance Dashboard | Comprehensive performance overview |
+| Category | Assets |
+|----------|--------|
+| Tokenizers | BPE at 8k, 16k, 32k, 64k vocab sizes |
+| N-gram models | 2 / 3 / 4 / 5-gram (word & subword) |
+| Markov chains | Context 1–5 (word & subword) |
+| Embeddings | 32d, 64d, 128d — mono & aligned |
+| Vocabulary | Full frequency list + Zipf analysis |
+| Statistics | Corpus & model statistics JSON |
+
+## Metrics Summary
+
+| Component | Model | Key Metric | Value |
+|-----------|-------|------------|-------|
+| Tokenizer | 8k BPE | Compression | 3.02x |
+| Tokenizer | 16k BPE | Compression | 3.30x |
+| Tokenizer | 32k BPE | Compression | 3.56x |
+| Tokenizer | 64k BPE | Compression | 3.82x 🏆 |
+| N-gram | 2-gram (subword) | Perplexity | 255 🏆 |
+| N-gram | 2-gram (word) | Perplexity | 1,027 |
+| N-gram | 3-gram (subword) | Perplexity | 1,284 |
+| N-gram | 3-gram (word) | Perplexity | 1,698 |
+| N-gram | 4-gram (subword) | Perplexity | 3,345 |
+| N-gram | 4-gram (word) | Perplexity | 3,109 |
+| N-gram | 5-gram (subword) | Perplexity | 5,689 |
+| N-gram | 5-gram (word) | Perplexity | 3,900 |
+| Markov | ctx-1 (subword) | Predictability | 0.0% |
+| Markov | ctx-1 (word) | Predictability | 36.7% |
+| Markov | ctx-2 (subword) | Predictability | 0.0% |
+| Markov | ctx-2 (word) | Predictability | 74.0% |
+| Markov | ctx-3 (subword) | Predictability | 17.0% |
+| Markov | ctx-3 (word) | Predictability | 91.6% |
+| Markov | ctx-4 (subword) | Predictability | 43.6% |
+| Markov | ctx-4 (word) | Predictability | 95.2% 🏆 |
+| Vocabulary | full | Size | 31,623 |
+| Vocabulary | full | Zipf R² | 0.9880 |
+| Embeddings | mono_32d | Isotropy | 0.6948 |
+| Embeddings | mono_64d | Isotropy | 0.5226 |
+| Embeddings | mono_128d | Isotropy | 0.2352 |
+| Embeddings | aligned_32d | Isotropy | 0.6948 🏆 |
+| Embeddings | aligned_64d | Isotropy | 0.5226 |
+| Embeddings | aligned_128d | Isotropy | 0.2352 |
+| Alignment | aligned_32d | R@1 / R@5 / R@10 | 0.6% / 2.0% / 5.4% |
+| Alignment | aligned_64d | R@1 / R@5 / R@10 | 2.4% / 8.0% / 12.8% |
+| Alignment | aligned_128d | R@1 / R@5 / R@10 | 3.6% / 11.2% / 17.8% 🏆 |
+
+📊 **[Full ablation study, per-model breakdowns, and interpretation guide →](RESEARCH_REPORT.md)**
 
 ---
-## About This Project
-
-### Data Source
 
-Models trained on [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly) - a monthly snapshot of Wikipedia articles across 300+ languages.
+## About
 
-### Project
+Trained on [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly) — monthly snapshots of 300+ Wikipedia languages.
 
-A project by **[Wikilangs](https://wikilangs.org)** - Open-source NLP models for every Wikipedia language.
-
-### Maintainer
-
-[Omar Kamali](https://omarkamali.com) - [Omneity Labs](https://omneitylabs.com)
+A project by **[Wikilangs](https://wikilangs.org)** · Maintainer: [Omar Kamali](https://omarkamali.com) · [Omneity Labs](https://omneitylabs.com)
 
 ### Citation
 
-If you use these models in your research, please cite:
-
 ```bibtex
 @misc{wikilangs2025,
-  author = {Kamali, Omar},
-  title = {Wikilangs: Open NLP Models for Wikipedia Languages},
-  year = {2025},
-  doi = {10.5281/zenodo.18073153},
+  author    = {Kamali, Omar},
+  title     = {Wikilangs: Open NLP Models for Wikipedia Languages},
+  year      = {2025},
+  doi       = {10.5281/zenodo.18073153},
   publisher = {Zenodo},
-  url = {https://huggingface.co/wikilangs}
+  url       = {https://huggingface.co/wikilangs},
   institution = {Omneity Labs}
 }
 ```
 
-### License
-
-MIT License - Free for academic and commercial use.
-
 ### Links
 
-- 🌐 Website: [wikilangs.org](https://wikilangs.org)
-- 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
-- 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
-- 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
+- 🌐 [wikilangs.org](https://wikilangs.org)
+- 🌍 [Language page](https://wikilangs.org/languages/shi/)
+- 🎮 [Playground](https://wikilangs.org/playground/?lang=shi)
+- 🤗 [HuggingFace models](https://huggingface.co/wikilangs)
+- 📊 [wikipedia-monthly dataset](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
+- 👤 [Omar Kamali](https://huggingface.co/omarkamali)
 - 🤝 Sponsor: [Featherless AI](https://featherless.ai)
----
-*Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2026-01-10 20:02:34*
+**License:** MIT — free for academic and commercial use.
+
+---
+*Generated by Wikilangs Pipeline · 2026-03-02 12:00:32*

RESEARCH_REPORT.md

@@ -0,0 +1,686 @@
+# Tachelhit — Full Ablation Study & Research Report
+
+Detailed evaluation of all model variants trained on **Tachelhit** Wikipedia data by [Wikilangs](https://wikilangs.org).
+
+👈 [Back to README](README.md)
+
+## 📋 Repository Contents
+
+### Models & Assets
+
+- Tokenizers (8k, 16k, 32k, 64k)
+- N-gram models (2, 3, 4, 5-gram)
+- Markov chains (context of 1, 2, 3, 4 and 5)
+- Subword N-gram and Markov chains
+- Embeddings in various sizes and dimensions (aligned and unaligned)
+- Language Vocabulary
+- Language Statistics
+
+![Performance Dashboard](visualizations/performance_dashboard.png)
+
+### Analysis and Evaluation
+
+- [1. Tokenizer Evaluation](#1-tokenizer-evaluation)
+- [2. N-gram Model Evaluation](#2-n-gram-model-evaluation)
+- [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
+- [4. Vocabulary Analysis](#4-vocabulary-analysis)
+- [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
+- [6. Morphological Analysis (Experimental)](#6--morphological-analysis-experimental)
+- [7. Summary & Recommendations](#7-summary--recommendations)
+- [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
+- [Visualizations Index](#visualizations-index)
+
+---
+## 1. Tokenizer Evaluation
+
+![Tokenizer Compression](visualizations/tokenizer_compression.png)
+
+![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+
+![Tokenizer OOV](visualizations/tokenizer_oov.png)
+
+![Total Tokens](visualizations/tokenizer_total_tokens.png)
+
+### Results
+
+| Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
+|------------|-------------|---------------|----------|--------------|
+| **8k** | 3.017x | 3.02 | 1.3938% | 408,453 |
+| **16k** | 3.301x | 3.30 | 1.5252% | 373,263 |
+| **32k** | 3.557x | 3.56 | 1.6432% | 346,460 |
+| **64k** | 3.819x 🏆 | 3.82 | 1.7643% | 322,672 |
+
+### Tokenization Examples
+
+Below are sample sentences tokenized with each vocabulary size:
+
+**Sample 1:** `Sstekk iga yan ugḍiḍ imẓẓin. Assaɣ Tuzduɣt Tasnalɣa (morphologie) Tisaɣulin Msmu...`
+
+| Vocab | Tokens | Count |
+|-------|--------|-------|
+| 8k | `▁s ste kk ▁iga ▁yan ▁ugḍiḍ ▁imẓẓin . ▁assaɣ ▁tuzduɣt ... (+19 more)` | 29 |
+| 16k | `▁s ste kk ▁iga ▁yan ▁ugḍiḍ ▁imẓẓin . ▁assaɣ ▁tuzduɣt ... (+19 more)` | 29 |
+| 32k | `▁s stekk ▁iga ▁yan ▁ugḍiḍ ▁imẓẓin . ▁assaɣ ▁tuzduɣt ▁tasnalɣa ... (+18 more)` | 28 |
+| 64k | `▁sstekk ▁iga ▁yan ▁ugḍiḍ ▁imẓẓin . ▁assaɣ ▁tuzduɣt ▁tasnalɣa ▁( ... (+17 more)` | 27 |
+
+**Sample 2:** `Asimwas iga ass wiss Smmus g ussan n imalass. Tisaɣulin`
+
+| Vocab | Tokens | Count |
+|-------|--------|-------|
+| 8k | `▁as im was ▁iga ▁ass ▁wiss ▁smmus ▁g ▁ussan ▁n ... (+3 more)` | 13 |
+| 16k | `▁as imwas ▁iga ▁ass ▁wiss ▁smmus ▁g ▁ussan ▁n ▁imalass ... (+2 more)` | 12 |
+| 32k | `▁asimwas ▁iga ▁ass ▁wiss ▁smmus ▁g ▁ussan ▁n ▁imalass . ... (+1 more)` | 11 |
+| 64k | `▁asimwas ▁iga ▁ass ▁wiss ▁smmus ▁g ▁ussan ▁n ▁imalass . ... (+1 more)` | 11 |
+
+**Sample 3:** `Turdut (S turdut: اردو ) tga tutlayt nna s sawaln ayt Bakistan d Lhnd. Isuɣal`
+
+| Vocab | Tokens | Count |
+|-------|--------|-------|
+| 8k | `▁tur dut ▁( s ▁tur dut : ▁ا ر دو ... (+14 more)` | 24 |
+| 16k | `▁tur dut ▁( s ▁tur dut : ▁ار دو ▁) ... (+13 more)` | 23 |
+| 32k | `▁turdut ▁( s ▁turdut : ▁اردو ▁) ▁tga ▁tutlayt ▁nna ... (+9 more)` | 19 |
+| 64k | `▁turdut ▁( s ▁turdut : ▁اردو ▁) ▁tga ▁tutlayt ▁nna ... (+8 more)` | 18 |
+
+
+### Key Findings
+
+- **Best Compression:** 64k achieves 3.819x compression
+- **Lowest UNK Rate:** 8k with 1.3938% unknown tokens
+- **Trade-off:** Larger vocabularies improve compression but increase model size
+- **Recommendation:** 32k vocabulary provides optimal balance for production use
+
+---
+## 2. N-gram Model Evaluation
+
+![N-gram Perplexity](visualizations/ngram_perplexity.png)
+
+![N-gram Unique](visualizations/ngram_unique.png)
+
+![N-gram Coverage](visualizations/ngram_coverage.png)
+
+### Results
+
+| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|---------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | Word | 1,027 | 10.00 | 23,244 | 45.7% | 81.7% |
+| **2-gram** | Subword | 255 🏆 | 7.99 | 3,782 | 68.8% | 99.0% |
+| **3-gram** | Word | 1,698 | 10.73 | 46,062 | 39.0% | 76.4% |
+| **3-gram** | Subword | 1,284 | 10.33 | 29,101 | 35.1% | 84.7% |
+| **4-gram** | Word | 3,109 | 11.60 | 90,318 | 35.2% | 68.9% |
+| **4-gram** | Subword | 3,345 | 11.71 | 117,821 | 23.5% | 73.6% |
+| **5-gram** | Word | 3,900 | 11.93 | 100,607 | 35.2% | 65.7% |
+| **5-gram** | Subword | 5,689 | 12.47 | 238,898 | 18.6% | 68.5% |
+
+### Top 5 N-grams by Size
+
+**2-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `tgmiḍi n` | 30,047 |
+| 2 | `n usggʷas` | 27,406 |
+| 3 | `umḍan n` | 26,921 |
+| 4 | `n imzdaɣn` | 25,250 |
+| 5 | `tlkm tgmiḍi` | 24,096 |
+
+**3-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `tlkm tgmiḍi n` | 24,096 |
+| 2 | `tamattayt n usɣiws` | 16,122 |
+| 3 | `tasmirit tamattayt n` | 15,740 |
+| 4 | `umḍan n imzdaɣn` | 14,946 |
+| 5 | `g tlkm tgmiḍi` | 12,050 |
+
+**4-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `tasmirit tamattayt n usɣiws` | 15,739 |
+| 2 | `g tlkm tgmiḍi n` | 12,050 |
+| 3 | `ad i trfiqt n` | 8,924 |
+| 4 | `uḍwwaṛ ad i trfiqt` | 8,917 |
+| 5 | `umḍan n imzdaɣn nns` | 8,916 |
+
+**5-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `uḍwwaṛ ad i trfiqt n` | 8,916 |
+| 2 | `n imzdaɣn tasmirit tamattayt n` | 8,910 |
+| 3 | `amatay n imzdaɣn tasmirit tamattayt` | 8,910 |
+| 4 | `imzdaɣn tasmirit tamattayt n usɣiws` | 8,910 |
+| 5 | `ilkm umḍan n imzdaɣn nns` | 8,904 |
+
+**2-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `n _` | 653,950 |
+| 2 | `_ n` | 401,960 |
+| 3 | `_ t` | 358,450 |
+| 4 | `_ i` | 253,361 |
+| 5 | `t a` | 205,185 |
+
+**3-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `_ n _` | 294,525 |
+| 2 | `_ t a` | 132,562 |
+| 3 | `n _ t` | 104,642 |
+| 4 | `a n _` | 103,515 |
+| 5 | `_ ɣ _` | 101,882 |
+
+**4-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `_ n _ u` | 84,436 |
+| 2 | `t _ n _` | 67,385 |
+| 3 | `_ n _ i` | 61,498 |
+| 4 | `_ n _ t` | 56,134 |
+| 5 | `n _ u s` | 52,239 |
+
+**5-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `_ n _ u s` | 51,413 |
+| 2 | `m z d a ɣ` | 46,710 |
+| 3 | `g g ʷ a s` | 34,963 |
+| 4 | `s g g ʷ a` | 34,938 |
+| 5 | `_ n n a _` | 34,315 |
+
+
+### Key Findings
+
+- **Best Perplexity:** 2-gram (subword) with 255
+- **Entropy Trend:** Decreases with larger n-grams (more predictable)
+- **Coverage:** Top-1000 patterns cover ~68% of corpus
+- **Recommendation:** 4-gram or 5-gram for best predictive performance
+
+---
+## 3. Markov Chain Evaluation
+
+![Markov Entropy](visualizations/markov_entropy.png)
+
+![Markov Contexts](visualizations/markov_contexts.png)
+
+![Markov Branching](visualizations/markov_branching.png)
+
+### Results
+
+| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | Word | 0.6330 | 1.551 | 4.06 | 76,272 | 36.7% |
+| **1** | Subword | 1.2927 | 2.450 | 10.38 | 804 | 0.0% |
+| **2** | Word | 0.2598 | 1.197 | 1.65 | 308,953 | 74.0% |
+| **2** | Subword | 1.0716 | 2.102 | 6.52 | 8,341 | 0.0% |
+| **3** | Word | 0.0840 | 1.060 | 1.19 | 508,729 | 91.6% |
+| **3** | Subword | 0.8300 | 1.778 | 3.82 | 54,358 | 17.0% |
+| **4** | Word | 0.0475 🏆 | 1.033 | 1.13 | 601,513 | 95.2% |
+| **4** | Subword | 0.5642 | 1.479 | 2.43 | 207,789 | 43.6% |
+
+### Generated Text Samples (Word-based)
+
+Below are text samples generated from each word-based Markov chain model:
+
+**Context Size 1:**
+
+1. `n twtmin ɣ tsga n lfṛaṛḥa nna mi ilkm umḍan n ayt ɛli n tarskkilt 43`
+2. `ɣ tmnaḍt n urtzaɣ taḍwwaṛḍt n tarwuri 2 aslmd g tlkm tgmiḍi n ism n isrɣinn`
+3. `d ublulls dar gr d lli tmmal tflwit yaḍn ngr adrar n bni matar m sidi`
+
+**Context Size 2:**
+
+1. `tgmiḍi n tarskkilt 70 82 gr mddn nna dar gr 6 d 11 n usggʷas niɣ uggar`
+2. `n usggʷas 28 48 dar tsdnan 3 5 aslmd g tlkm tgmiḍi n 35 1 ig unammas`
+3. `umḍan n imzdaɣn n lmɣrib ɣ tsga n trudant n fas amknas ɣ lmɣrib iḍfaṛ uḍwwaṛ ad`
+
+**Context Size 3:**
+
+1. `tlkm tgmiḍi n uslmd 91 97 gr irban d trbatin nna dar gr 6 d 11 n usggʷas`
+2. `tamattayt n usɣiws aṛcif 14 ɣuct tisnaddadin tisnaddadin timatayin iggʷiz umḍan n imzdaɣn n tamyawas...`
+3. `tasmirit tamattayt n usɣiws tisaɣulin isɣwan yaḍnin tasmirit tamattayt n usɣiws ɣ iga umḍan n imawaḍ...`
+
+**Context Size 4:**
+
+1. `tasmirit tamattayt n usɣiws aṛcif 14 ɣuct tisnaddadin tisnaddadin timatayin iɣli umḍan n imzdaɣn n a...`
+2. `g tlkm tgmiḍi n uslmd 98 7 gr irban d trbatin nna dar gr 6 d 11 n usggʷas`
+3. `ad i trfiqt n ifrdaw tiɣanimin nna ɣ llan 20 n iḍuṛan ilkm umḍan n imzdaɣn nns 997 n`
+
+
+### Generated Text Samples (Subword-based)
+
+Below are text samples generated from each subword-based Markov chain model:
+
+**Context Size 1:**
+
+1. `_sgmawimda_tabir`
+2. `an_4422._uwafarg`
+3. `n),_nartan_49_an`
+
+**Context Size 2:**
+
+1. `n_des_ig_twuṭṭa_u`
+2. `_n_10.09_n_uḍwwaṛ`
+3. `_tɛṛanbattamaslmd`
+
+**Context Size 3:**
+
+1. `_n_umḍan_d_imir_an`
+2. `_tamatay_n_i_tugt_`
+3. `n_tznit_taru_260_n`
+
+**Context Size 4:**
+
+1. `_n_umzdaɣn_tasga_n_`
+2. `t_n_ayt_baha_ɣ_lli_`
+3. `_n_iɣ_isggʷasn_d_tr`
+
+
+### Key Findings
+
+- **Best Predictability:** Context-4 (word) with 95.2% predictability
+- **Branching Factor:** Decreases with context size (more deterministic)
+- **Memory Trade-off:** Larger contexts require more storage (207,789 contexts)
+- **Recommendation:** Context-3 or Context-4 for text generation
+
+---
+## 4. Vocabulary Analysis
+
+![Zipf's Law](visualizations/zipf_law.png)
+
+![Top Words](visualizations/top20_words.png)
+
+![Coverage Curve](visualizations/vocab_coverage.png)
+
+### Statistics
+
+| Metric | Value |
+|--------|-------|
+| Vocabulary Size | 31,623 |
+| Total Tokens | 2,378,986 |
+| Mean Frequency | 75.23 |
+| Median Frequency | 4 |
+| Frequency Std Dev | 1969.53 |
+
+### Most Common Words
+
+| Rank | Word | Frequency |
+|------|------|-----------|
+| 1 | n | 294,723 |
+| 2 | ɣ | 102,005 |
+| 3 | d | 64,397 |
+| 4 | s | 35,003 |
+| 5 | nna | 34,361 |
+| 6 | imzdaɣn | 31,398 |
+| 7 | dar | 30,865 |
+| 8 | gr | 30,722 |
+| 9 | tgmiḍi | 30,050 |
+| 10 | usggʷas | 28,210 |
+
+### Least Common Words (from vocabulary)
+
+| Rank | Word | Frequency |
+|------|------|-----------|
+| 1 | tdarwinit | 2 |
+| 2 | talmuqqdimt | 2 |
+| 3 | ttawnn | 2 |
+| 4 | taggrgist | 2 |
+| 5 | umdgar | 2 |
+| 6 | uqṛiḍ | 2 |
+| 7 | dearborn | 2 |
+| 8 | ghosts | 2 |
+| 9 | tremblay | 2 |
+| 10 | tmmndl | 2 |
+
+### Zipf's Law Analysis
+
+| Metric | Value |
+|--------|-------|
+| Zipf Coefficient | 1.2850 |
+| R² (Goodness of Fit) | 0.988028 |
+| Adherence Quality | **excellent** |
+
+### Coverage Analysis
+
+| Top N Words | Coverage |
+|-------------|----------|
+| Top 100 | 69.6% |
+| Top 1,000 | 90.6% |
+| Top 5,000 | 95.5% |
+| Top 10,000 | 97.3% |
+
+### Key Findings
+
+- **Zipf Compliance:** R²=0.9880 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 69.6% of corpus
+- **Long Tail:** 21,623 words needed for remaining 2.7% coverage
+
+---
+## 5. Word Embeddings Evaluation
+
+![Embedding Isotropy](visualizations/embedding_isotropy.png)
+
+![Similarity Matrix](visualizations/embedding_similarity.png)
+
+![t-SNE Words](visualizations/tsne_words.png)
+
+![t-SNE Sentences](visualizations/tsne_sentences.png)
+
+
+### 5.1 Cross-Lingual Alignment
+
+![Alignment Quality](visualizations/embedding_alignment_quality.png)
+
+![Multilingual t-SNE](visualizations/embedding_tsne_multilingual.png)
+
+
+### 5.2 Model Comparison
+
+| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
+|-------|-----------|----------|------------------|---------------|----------------|
+| **mono_32d** | 32 | 0.6948 | 0.3782 | N/A | N/A |
+| **mono_64d** | 64 | 0.5226 | 0.3533 | N/A | N/A |
+| **mono_128d** | 128 | 0.2352 | 0.3437 | N/A | N/A |
+| **aligned_32d** | 32 | 0.6948 🏆 | 0.3868 | 0.0060 | 0.0540 |
+| **aligned_64d** | 64 | 0.5226 | 0.3472 | 0.0240 | 0.1280 |
+| **aligned_128d** | 128 | 0.2352 | 0.3345 | 0.0360 | 0.1780 |
+
+### Key Findings
+
+- **Best Isotropy:** aligned_32d with 0.6948 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.3573. Lower values indicate better semantic separation.
+- **Alignment Quality:** Aligned models achieve up to 3.6% R@1 in cross-lingual retrieval.
+- **Recommendation:** 128d aligned for best cross-lingual performance
+
+---
+## 6.  Morphological Analysis (Experimental)
+
+This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
+
+### 6.1 Productivity & Complexity
+
+| Metric | Value | Interpretation | Recommendation |
+|--------|-------|----------------|----------------|
+| Productivity Index | **5.000** | High morphological productivity | Reliable analysis |
+| Idiomaticity Gap | **-0.041** | Low formulaic content | - |
+
+### 6.2 Affix Inventory (Productive Units)
+
+These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
+
+#### Productive Prefixes
+| Prefix | Examples |
+|--------|----------|
+| `-t` | tmurrant, taɣwwaɣt, tuwuri |
+| `-i` | ill, itturray, iɛisayn |
+| `-ta` | taɣwwaɣt, tagnsant, tabrruct |
+| `-a` | anmmassu, asaki, atayn |
+| `-u` | umzizwr, uɣnja, umdlu |
+| `-l` | lmuddn, lmɣrib, lbadiɛ |
+| `-ti` | timqqit, tisutam, tinglizt |
+| `-m` | mggrn, mennawt, magẓnt |
+
+#### Productive Suffixes
+| Suffix | Examples |
+|--------|----------|
+| `-n` | atayn, mggrn, krnun |
+| `-t` | tmurrant, priest, taɣwwaɣt |
+| `-a` | uɣnja, phoenicia, iɣrruba |
+| `-in` | ɛalawiyyin, bdrnin, irwin |
+| `-s` | ghosts, yuns, palmas |
+| `-i` | asaki, bani, tuwuri |
+| `-e` | became, institute, neige |
+| `-an` | dan, ubrkan, uljmɛan |
+
+### 6.3 Bound Stems (Lexical Roots)
+
+Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
+
+| Stem | Cohesion | Substitutability | Examples |
+|------|----------|------------------|----------|
+| `adda` | 1.68x | 52 contexts | addan, hadda, wadda |
+| `ggar` | 1.97x | 22 contexts | uggar, ggarn, iggar |
+| `ggʷa` | 1.62x | 43 contexts | ḥggʷa, aggʷa, zggʷar |
+| `ugga` | 1.91x | 21 contexts | uggar, uggan, tugga |
+| `wuri` | 1.70x | 30 contexts | twuri, tuwuri, twwuri |
+| `tion` | 2.05x | 14 contexts | nation, notion, action |
+| `matt` | 1.64x | 26 contexts | matta, nmatti, amattu |
+| `lati` | 1.61x | 27 contexts | latif, latin, talati |
+| `ɣrib` | 1.76x | 20 contexts | aɣrib, mɣrib, lmɣrib |
+| `mɣri` | 1.77x | 13 contexts | tmɣri, imɣri, mɣrib |
+| `ddad` | 1.62x | 14 contexts | ḥddad, addad, addadn |
+| `mata` | 1.54x | 14 contexts | smata, amata, umata |
+
+### 6.4 Affix Compatibility (Co-occurrence)
+
+This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
+
+| Prefix | Suffix | Frequency | Examples |
+|--------|--------|-----------|----------|
+| `-t` | `-t` | 636 words | takrrayt, tusnaktant |
+| `-i` | `-n` | 489 words | ittajjan, igatn |
+| `-t` | `-n` | 323 words | tunisian, tmttawin |
+| `-t` | `-in` | 264 words | tmttawin, tmdinin |
+| `-l` | `-a` | 101 words | lqliɛa, lɛmaṛa |
+| `-t` | `-a` | 71 words | tggʷra, tawayya |
+| `-i` | `-an` | 60 words | ittajjan, ixxan |
+| `-a` | `-n` | 60 words | aẓuran, ayncṭayn |
+| `-l` | `-t` | 39 words | lmɛiṭat, luṭilat |
+| `-a` | `-an` | 39 words | aẓuran, alilan |
+
+### 6.5 Recursive Morpheme Segmentation
+
+Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
+
+| Word | Suggested Split | Confidence | Stem |
+|------|-----------------|------------|------|
+| magrebini | **`magreb-in-i`** | 7.5 | `in` |
+| mzaraynin | **`mzaray-n-in`** | 7.5 | `n` |
+| imaynutnin | **`imaynut-n-in`** | 7.5 | `n` |
+| tiɣrmanin | **`tiɣrm-an-in`** | 7.5 | `an` |
+| ikkattinn | **`ikkatt-in-n`** | 7.5 | `in` |
+| tisntutin | **`tisntu-t-in`** | 7.5 | `t` |
+| tasnmḍant | **`tasnmḍ-an-t`** | 7.5 | `an` |
+| tuɣnijinin | **`tuɣnij-in-in`** | 7.5 | `in` |
+| ittyawnna | **`ittyaw-n-na`** | 7.5 | `n` |
+| tinidlisn | **`t-in-idlisn`** | 7.5 | `idlisn` |
+| fransisku | **`fransis-k-u`** | 7.5 | `k` |
+| gibraltar | **`gibral-t-ar`** | 7.5 | `t` |
+| iblḥsanin | **`iblḥsa-n-in`** | 7.5 | `n` |
+| ittyurnan | **`ittyur-n-an`** | 7.5 | `n` |
+| africaines | **`africa-in-es`** | 7.5 | `in` |
+
+### 6.6 Linguistic Interpretation
+
+> **Automated Insight:**
+The language Tachelhit shows high morphological productivity. The subword models are significantly more efficient than word models, suggesting a rich system of affixation or compounding.
+
+---
+## 7. Summary & Recommendations
+
+![Performance Dashboard](visualizations/performance_dashboard.png)
+
+### Production Recommendations
+
+| Component | Recommended | Rationale |
+|-----------|-------------|-----------|
+| Tokenizer | **64k BPE** | Best compression (3.82x) |
+| N-gram | **2-gram** | Lowest perplexity (255) |
+| Markov | **Context-4** | Highest predictability (95.2%) |
+| Embeddings | **100d** | Balanced semantic capture and isotropy |
+
+
+---
+## Appendix: Metrics Glossary & Interpretation Guide
+
+This section provides definitions, intuitions, and guidance for interpreting the metrics used throughout this report.
+
+### Tokenizer Metrics
+
+**Compression Ratio**
+> *Definition:* The ratio of characters to tokens (chars/token). Measures how efficiently the tokenizer represents text.
+>
+> *Intuition:* Higher compression means fewer tokens needed to represent the same text, reducing sequence lengths for downstream models. A 3x compression means ~3 characters per token on average.
+>
+> *What to seek:* Higher is generally better for efficiency, but extremely high compression may indicate overly aggressive merging that loses morphological information.
+
+**Average Token Length (Fertility)**
+> *Definition:* Mean number of characters per token produced by the tokenizer.
+>
+> *Intuition:* Reflects the granularity of tokenization. Longer tokens capture more context but may struggle with rare words; shorter tokens are more flexible but increase sequence length.
+>
+> *What to seek:* Balance between 2-5 characters for most languages. Arabic/morphologically-rich languages may benefit from slightly longer tokens.
+
+**Unknown Token Rate (OOV Rate)**
+> *Definition:* Percentage of tokens that map to the unknown/UNK token, indicating words the tokenizer cannot represent.
+>
+> *Intuition:* Lower OOV means better vocabulary coverage. High OOV indicates the tokenizer encounters many unseen character sequences.
+>
+> *What to seek:* Below 1% is excellent; below 5% is acceptable. BPE tokenizers typically achieve very low OOV due to subword fallback.
+
+### N-gram Model Metrics
+
+**Perplexity**
+> *Definition:* Measures how "surprised" the model is by test data. Mathematically: 2^(cross-entropy). Lower values indicate better prediction.
+>
+> *Intuition:* If perplexity is 100, the model is as uncertain as if choosing uniformly among 100 options at each step. A perplexity of 10 means effectively choosing among 10 equally likely options.
+>
+> *What to seek:* Lower is better. Perplexity decreases with larger n-grams (more context). Values vary widely by language and corpus size.
+
+**Entropy**
+> *Definition:* Average information content (in bits) needed to encode the next token given the context. Related to perplexity: perplexity = 2^entropy.
+>
+> *Intuition:* High entropy means high uncertainty/randomness; low entropy means predictable patterns. Natural language typically has entropy between 1-4 bits per character.
+>
+> *What to seek:* Lower entropy indicates more predictable text patterns. Entropy should decrease as n-gram size increases.
+
+**Coverage (Top-K)**
+> *Definition:* Percentage of corpus occurrences explained by the top K most frequent n-grams.
+>
+> *Intuition:* High coverage with few patterns indicates repetitive/formulaic text; low coverage suggests diverse vocabulary usage.
+>
+> *What to seek:* Depends on use case. For language modeling, moderate coverage (40-60% with top-1000) is typical for natural text.
+
+### Markov Chain Metrics
+
+**Average Entropy**
+> *Definition:* Mean entropy across all contexts, measuring average uncertainty in next-word prediction.
+>
+> *Intuition:* Lower entropy means the model is more confident about what comes next. Context-1 has high entropy (many possible next words); Context-4 has low entropy (few likely continuations).
+>
+> *What to seek:* Decreasing entropy with larger context sizes. Very low entropy (<0.1) indicates highly deterministic transitions.
+
+**Branching Factor**
+> *Definition:* Average number of unique next tokens observed for each context.
+>
+> *Intuition:* High branching = many possible continuations (flexible but uncertain); low branching = few options (predictable but potentially repetitive).
+>
+> *What to seek:* Branching factor should decrease with context size. Values near 1.0 indicate nearly deterministic chains.
+
+**Predictability**
+> *Definition:* Derived metric: (1 - normalized_entropy) × 100%. Indicates how deterministic the model's predictions are.
+>
+> *Intuition:* 100% predictability means the next word is always certain; 0% means completely random. Real text falls between these extremes.
+>
+> *What to seek:* Higher predictability for text generation quality, but too high (>98%) may produce repetitive output.
+
+### Vocabulary & Zipf's Law Metrics
+
+**Zipf's Coefficient**
+> *Definition:* The slope of the log-log plot of word frequency vs. rank. Zipf's law predicts this should be approximately -1.
+>
+> *Intuition:* A coefficient near -1 indicates the corpus follows natural language patterns where a few words are very common and most words are rare.
+>
+> *What to seek:* Values between -0.8 and -1.2 indicate healthy natural language distribution. Deviations may suggest domain-specific or artificial text.
+
+**R² (Coefficient of Determination)**
+> *Definition:* Measures how well the linear fit explains the frequency-rank relationship. Ranges from 0 to 1.
+>
+> *Intuition:* R² near 1.0 means the data closely follows Zipf's law; lower values indicate deviation from expected word frequency patterns.
+>
+> *What to seek:* R² > 0.95 is excellent; > 0.99 indicates near-perfect Zipf adherence typical of large natural corpora.
+
+**Vocabulary Coverage**
+> *Definition:* Cumulative percentage of corpus tokens accounted for by the top N words.
+>
+> *Intuition:* Shows how concentrated word usage is. If top-100 words cover 50% of text, the corpus relies heavily on common words.
+>
+> *What to seek:* Top-100 covering 30-50% is typical. Higher coverage indicates more repetitive text; lower suggests richer vocabulary.
+
+### Word Embedding Metrics
+
+**Isotropy**
+> *Definition:* Measures how uniformly distributed vectors are in the embedding space. Computed as the ratio of minimum to maximum singular values.
+>
+> *Intuition:* High isotropy (near 1.0) means vectors spread evenly in all directions; low isotropy means vectors cluster in certain directions, reducing expressiveness.
+>
+> *What to seek:* Higher isotropy generally indicates better-quality embeddings. Values > 0.1 are reasonable; > 0.3 is good. Lower-dimensional embeddings tend to have higher isotropy.
+
+**Average Norm**
+> *Definition:* Mean magnitude (L2 norm) of word vectors in the embedding space.
+>
+> *Intuition:* Indicates the typical "length" of vectors. Consistent norms suggest stable training; high variance may indicate some words are undertrained.
+>
+> *What to seek:* Relatively consistent norms across models. The absolute value matters less than consistency (low std deviation).
+
+**Cosine Similarity**
+> *Definition:* Measures angular similarity between vectors, ranging from -1 (opposite) to 1 (identical direction).
+>
+> *Intuition:* Words with similar meanings should have high cosine similarity. This is the standard metric for semantic relatedness in embeddings.
+>
+> *What to seek:* Semantically related words should score > 0.5; unrelated words should be near 0. Synonyms often score > 0.7.
+
+**t-SNE Visualization**
+> *Definition:* t-Distributed Stochastic Neighbor Embedding - a dimensionality reduction technique that preserves local structure for visualization.
+>
+> *Intuition:* Clusters in t-SNE plots indicate groups of semantically related words. Spread indicates vocabulary diversity; tight clusters suggest semantic coherence.
+>
+> *What to seek:* Meaningful clusters (e.g., numbers together, verbs together). Avoid over-interpreting distances - t-SNE preserves local, not global, structure.
+
+### General Interpretation Guidelines
+
+1. **Compare within model families:** Metrics are most meaningful when comparing models of the same type (e.g., 8k vs 64k tokenizer).
+2. **Consider trade-offs:** Better performance on one metric often comes at the cost of another (e.g., compression vs. OOV rate).
+3. **Context matters:** Optimal values depend on downstream tasks. Text generation may prioritize different metrics than classification.
+4. **Corpus influence:** All metrics are influenced by corpus characteristics. Wikipedia text differs from social media or literature.
+5. **Language-specific patterns:** Morphologically rich languages (like Arabic) may show different optimal ranges than analytic languages.
+
+
+### Visualizations Index
+
+| Visualization | Description |
+|---------------|-------------|
+| Tokenizer Compression | Compression ratios by vocabulary size |
+| Tokenizer Fertility | Average token length by vocabulary |
+| Tokenizer OOV | Unknown token rates |
+| Tokenizer Total Tokens | Total tokens by vocabulary |
+| N-gram Perplexity | Perplexity by n-gram size |
+| N-gram Entropy | Entropy by n-gram size |
+| N-gram Coverage | Top pattern coverage |
+| N-gram Unique | Unique n-gram counts |
+| Markov Entropy | Entropy by context size |
+| Markov Branching | Branching factor by context |
+| Markov Contexts | Unique context counts |
+| Zipf's Law | Frequency-rank distribution with fit |
+| Vocab Frequency | Word frequency distribution |
+| Top 20 Words | Most frequent words |
+| Vocab Coverage | Cumulative coverage curve |
+| Embedding Isotropy | Vector space uniformity |
+| Embedding Norms | Vector magnitude distribution |
+| Embedding Similarity | Word similarity heatmap |
+| Nearest Neighbors | Similar words for key terms |
+| t-SNE Words | 2D word embedding visualization |
+| t-SNE Sentences | 2D sentence embedding visualization |
+| Position Encoding | Encoding method comparison |
+| Model Sizes | Storage requirements |
+| Performance Dashboard | Comprehensive performance overview |
+
+---
+👈 [Back to README](README.md)
+
+*Generated by Wikilangs Pipeline · 2026-03-02 12:00:43*

models/embeddings/aligned/shi_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5180c0aea685af906cddd6c7f78d295251ff71d075389889cbfaca179459d97d
-size 1041689156
+oid sha256:43f1f49436ae92b235449641d5ca8a09092a5b10bb27be36dd23f4e928d75ed2
+size 1041700607

models/embeddings/aligned/shi_128d.projection.npy

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:211531ea12a8ff3821d927f750a97d85b2423306e1fdfa1562cf211f382f3430
+oid sha256:c4cde5d426b37eed3fc36f35779a7f5d7747bbac7c622b1b0c8dac9abf2566db
 size 65664

models/embeddings/aligned/shi_128d_metadata.json

@@ -3,6 +3,6 @@
   "dimension": 128,
   "version": "aligned",
   "hub_language": "en",
-  "seed_vocab_size": 4523,
-  "vocab_size": 17009
+  "seed_vocab_size": 4520,
+  "vocab_size": 17020
 }

models/embeddings/aligned/shi_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e4451f5a13fa057c50ce53cba54ab91900310da2c07c02d2c2368d9125262b4a
-size 260626244
+oid sha256:a8cfecd35338c94847992377475ab7c9f94676c0ea3fc71a13cafdfdd52d47cf
+size 260629247

models/embeddings/aligned/shi_32d.projection.npy

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0a8ce5fe0b0b2c491a3310da67bdd0a76624779c9125a008e80bdd50e35e3b43
+oid sha256:d5d7af1086a96ecbb466e8b2dde410e31bf8771261fbb0872b1fac385f32fe5f
 size 4224

models/embeddings/aligned/shi_32d_metadata.json

@@ -3,6 +3,6 @@
   "dimension": 32,
   "version": "aligned",
   "hub_language": "en",
-  "seed_vocab_size": 4523,
-  "vocab_size": 17009
+  "seed_vocab_size": 4520,
+  "vocab_size": 17020
 }

models/embeddings/aligned/shi_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:025ce2122ab99474ed6412bca9b451b0bdb3a7f03c5a46ae06f008b0a4d12955
-size 520980548
+oid sha256:a25dfe597a349af0cdd94e9615d327c26fbbdb6a8bd88d283c5a9a9b0c29c052
+size 520986367

models/embeddings/aligned/shi_64d.projection.npy

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:abdb81967fc19addd955605ef7f52d5cee989de32bc0f340a6d9a695222a7329
+oid sha256:d4bb82ee32094a7637efe4c90a00cc7c67176e37d61f0ba6681df1a06068ef7f
 size 16512

models/embeddings/aligned/shi_64d_metadata.json

@@ -3,6 +3,6 @@
   "dimension": 64,
   "version": "aligned",
   "hub_language": "en",
-  "seed_vocab_size": 4523,
-  "vocab_size": 17009
+  "seed_vocab_size": 4520,
+  "vocab_size": 17020
 }

models/embeddings/monolingual/shi_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5180c0aea685af906cddd6c7f78d295251ff71d075389889cbfaca179459d97d
-size 1041689156
+oid sha256:43f1f49436ae92b235449641d5ca8a09092a5b10bb27be36dd23f4e928d75ed2
+size 1041700607

models/embeddings/monolingual/shi_128d_metadata.json

@@ -10,7 +10,7 @@
     "epochs": 5,
     "encoding_method": "rope",
     "dim": 128,
-    "threads": 5
+    "threads": 40
   },
-  "vocab_size": 17009
+  "vocab_size": 17020
 }

models/embeddings/monolingual/shi_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e4451f5a13fa057c50ce53cba54ab91900310da2c07c02d2c2368d9125262b4a
-size 260626244
+oid sha256:a8cfecd35338c94847992377475ab7c9f94676c0ea3fc71a13cafdfdd52d47cf
+size 260629247

models/embeddings/monolingual/shi_32d_metadata.json

@@ -10,7 +10,7 @@
     "epochs": 5,
     "encoding_method": "rope",
     "dim": 32,
-    "threads": 5
+    "threads": 40
   },
-  "vocab_size": 17009
+  "vocab_size": 17020
 }

models/embeddings/monolingual/shi_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:025ce2122ab99474ed6412bca9b451b0bdb3a7f03c5a46ae06f008b0a4d12955
-size 520980548
+oid sha256:a25dfe597a349af0cdd94e9615d327c26fbbdb6a8bd88d283c5a9a9b0c29c052
+size 520986367

models/embeddings/monolingual/shi_64d_metadata.json

@@ -10,7 +10,7 @@
     "epochs": 5,
     "encoding_method": "rope",
     "dim": 64,
-    "threads": 5
+    "threads": 40
   },
-  "vocab_size": 17009
+  "vocab_size": 17020
 }

models/subword_markov/shi_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:965ea27e92e79f7c9fd28a1a56317eacd35ef7c8b8cf063aec63347b3487811e
-size 69971
+oid sha256:1ab5b6e6ac56df19fd3d56842b69a627e0a7405220c1c3eac0d438a28a494fde
+size 67897

models/subword_markov/shi_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "shi",
-  "unique_contexts": 803,
-  "total_transitions": 12293493
+  "unique_contexts": 804,
+  "total_transitions": 12295406
 }

models/subword_markov/shi_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:92cbe593aca2b2220d718c5ea3130f8a3d5f641e75cc72d54c6c3314ac3fcb8b
-size 431239
+oid sha256:f1a82548e589b92588b7ab34f6128cb60ca928c17fed55f4859fd13e4bf9a1aa
+size 436486

models/subword_markov/shi_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "shi",
-  "unique_contexts": 8338,
-  "total_transitions": 12282565
+  "unique_contexts": 8341,
+  "total_transitions": 12284478
 }

models/subword_markov/shi_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5673b994a73e3cada02d54601e98214c2f8b0812e5d2115fc3103ddb4faac86f
-size 1570124
+oid sha256:c3955386ff8b4c1aac6b7782eaad8ba2c65c8e45781ae53802d651a9d77c50ee
+size 1577233

models/subword_markov/shi_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "shi",
-  "unique_contexts": 54347,
-  "total_transitions": 12271637
+  "unique_contexts": 54358,
+  "total_transitions": 12273550
 }

models/subword_markov/shi_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:10caedfe683a0bebf723f8cf63c4075858f95ab8a974dcd83d66de06ad06f152
-size 4216173
+oid sha256:54c93f72e85ee610ce4794367342b4695d74e6b1151660668cf8e91d116ab64f
+size 4179316

models/subword_markov/shi_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "shi",
-  "unique_contexts": 207735,
-  "total_transitions": 12260709
+  "unique_contexts": 207789,
+  "total_transitions": 12262622
 }

models/subword_ngram/shi_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:264a2d107c0b01701792c13347869860c493b7f303e30529e9862b869d5dfb1f
-size 53113
+oid sha256:920be97eae2f72d66a70f66a47c0fbc6010e6353cd8443ec2e1879b4ab9f08c8
+size 51711

models/subword_ngram/shi_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "shi",
-  "unique_ngrams": 3781,
-  "total_ngrams": 12293493
+  "unique_ngrams": 3782,
+  "total_ngrams": 12295406
 }

models/subword_ngram/shi_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:98c76510d57a43bdf2439de67e8e6f8f2cf73322dc1873edbdd04abf2229fa64
-size 373157
+oid sha256:f51efe7dd30b56c5a847d8eb18e3f5926c058ede9a3b84113d23558b1dff8f5d
+size 371866

models/subword_ngram/shi_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "shi",
-  "unique_ngrams": 29091,
-  "total_ngrams": 12282565
+  "unique_ngrams": 29101,
+  "total_ngrams": 12284478
 }

models/subword_ngram/shi_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2ec66bc60651234ac0c7f486dbb8911f5660d92a9712705cbec166218b12ccf0
-size 1395732
+oid sha256:ece392c150db5e41042b1f1463131a384716098853e47907f14a05bcce3cea56
+size 1385345

models/subword_ngram/shi_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "shi",
-  "unique_ngrams": 117787,
-  "total_ngrams": 12271637
+  "unique_ngrams": 117821,
+  "total_ngrams": 12273550
 }

models/subword_ngram/shi_5gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c2d2e731209c07771e78cd8f123affb87ad6fad22719f14da00433985438e7cb
-size 2843510
+oid sha256:9b593a4e2a22f872f1b914cae01b5d194054be484613f24e2807e6819506df10
+size 2864409

models/subword_ngram/shi_5gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 5,
   "variant": "subword",
   "language": "shi",
-  "unique_ngrams": 238802,
-  "total_ngrams": 12260709
+  "unique_ngrams": 238898,
+  "total_ngrams": 12262622
 }

models/tokenizer/shi_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:bf55f93a5c168c699ea7b6920d0c1dce79dad5ed2ec6f0883f5334b8eaf01faf
-size 504034
+oid sha256:dea6c7b9bf128d7b9b3ad53e1f56f5d19cb4070b69daab29ff97cbd7915e176e
+size 503979

models/tokenizer/shi_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:95e152c6683730a7aaa3b25d2d05e5cdcea432d4e701868a0f437eaba784c3c3
-size 776917
+oid sha256:d45b96c86ae6e602649478ea13f05b3a0d80e2f383f38743a99288ffa460642f
+size 776921

models/tokenizer/shi_tokenizer_64k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:35d7cf73a1c5fa600c08b521908e695a23b70a8a247c9ce00bce001b99605e2a
-size 1372695
+oid sha256:337be644a11642d83988dd942335c62600744f149225c6df3298d487fe9c2c2b
+size 1372696

models/tokenizer/shi_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:12f10cfa65935d01c6f35027ff53d1eba0c95648fa8e5dc24ae25a0c155b0b1b
-size 371147
+oid sha256:2262014ef46938da292640461b5504105bb3a1d09d889c2fbb4b5c0f5601f266
+size 371126

models/vocabulary/shi_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8aa16b4ff5792ecc72682c80a38b44f25d7eae7c4e7a1a61f10f57f8795f9e12
-size 528736
+oid sha256:0cb46834f112c721e7d30eb95f6399832acb10c559bad1ec9a71ca3b444b73c0
+size 527429

models/vocabulary/shi_vocabulary_metadata.json

@@ -1,17 +1,17 @@
 {
   "language": "shi",
-  "vocabulary_size": 31610,
+  "vocabulary_size": 31623,
   "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.031469987327161844,
+    "type_token_ratio": 0.031480475198470415,
     "coverage": {
-      "top_100": 0.6830915617715233,
-      "top_1000": 0.8892152125227943,
-      "top_5000": 0.9379014423761386,
-      "top_10000": 0.9550817833419236
+      "top_100": 0.6830340546800894,
+      "top_1000": 0.8891742698339413,
+      "top_5000": 0.9378737372924679,
+      "top_10000": 0.9550576586412044
     },
-    "hapax_count": 44651,
-    "hapax_ratio": 0.5855024193231141,
+    "hapax_count": 44675,
+    "hapax_ratio": 0.5855330414951899,
     "total_documents": 10928
   }
 }

models/word_markov/shi_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a960641178f1c734df546734d77f572b5824eea890e57f5b33664ee18a1b81ff
-size 2594157
+oid sha256:a037b3539cc1a51be346e2090ddbfda4e732441f256bf314ba7eb87ee0ec50a5
+size 2608428

models/word_markov/shi_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "shi",
-  "unique_contexts": 76235,
-  "total_transitions": 2412365
+  "unique_contexts": 76272,
+  "total_transitions": 2412733
 }

models/word_markov/shi_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ca2b881ebda57ee862c4f305d0f27ec4c7e03aeb89cc61d3f40f1fff8de62788
-size 5760236
+oid sha256:a43fa252c83eb91d1f6923e27bbbc6e9067265eee7415ea83af7164014be8c11
+size 5736513

models/word_markov/shi_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "shi",
-  "unique_contexts": 308778,
-  "total_transitions": 2401437
+  "unique_contexts": 308953,
+  "total_transitions": 2401805
 }

models/word_markov/shi_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:25ee5eb4f0959a5d169b844d85d2a0f97c439371968c7791593a5b73f29e93dd
-size 8170854
+oid sha256:ec84c5a2aa31b94c03d73d8303a801f75656c1586c45343c9aabc2db871d3430
+size 8241476

models/word_markov/shi_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "shi",
-  "unique_contexts": 508428,
-  "total_transitions": 2390509
+  "unique_contexts": 508729,
+  "total_transitions": 2390877
 }

models/word_markov/shi_markov_ctx4_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:17c766958e099580b89478e9f2f89167c4f22bbe880d222bab71e9521d926882
-size 9985346
+oid sha256:95128d1594fb711dcfd8b65380855e86b339a7cbb5ebd05c929b3e239f2d1b08
+size 9975630