README.md · wikilangs/bpy at main

bpy / README.md

omarkamali

Upload all models and assets for bpy (latest)

67cb1a6 verified 3 months ago

preview code

raw

history blame contribute delete

35.4 kB

	---
	language: bpy
	language_name: Bishnupriya
	language_family: indoaryan_eastern
	tags:
	- wikilangs
	- nlp
	- tokenizer
	- embeddings
	- n-gram
	- markov
	- wikipedia
	- feature-extraction
	- sentence-similarity
	- tokenization
	- n-grams
	- markov-chain
	- text-mining
	- fasttext
	- babelvec
	- vocabulous
	- vocabulary
	- monolingual
	- family-indoaryan_eastern
	license: mit
	library_name: wikilangs
	pipeline_tag: text-generation
	datasets:
	- omarkamali/wikipedia-monthly
	dataset_info:
	name: wikipedia-monthly
	description: Monthly snapshots of Wikipedia articles across 300+ languages
	metrics:
	- name: best_compression_ratio
	type: compression
	value: 4.935
	- name: best_isotropy
	type: isotropy
	value: 0.6926
	- name: vocabulary_size
	type: vocab
	value: 0
	generated: 2026-01-03
	---

	# Bishnupriya - Wikilangs Models
	## Comprehensive Research Report & Full Ablation Study

	This repository contains NLP models trained and evaluated by Wikilangs, specifically on Bishnupriya Wikipedia data.
	We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and word embeddings.

	## 📋 Repository Contents

	### Models & Assets

	- Tokenizers (8k, 16k, 32k, 64k)
	- N-gram models (2, 3, 4, 5-gram)
	- Markov chains (context of 1, 2, 3, 4 and 5)
	- Subword N-gram and Markov chains
	- Embeddings in various sizes and dimensions (aligned and unaligned)
	- Language Vocabulary
	- Language Statistics

	![Performance Dashboard](visualizations/performance_dashboard.png)

	### Analysis and Evaluation

	- [1. Tokenizer Evaluation](#1-tokenizer-evaluation)
	- [2. N-gram Model Evaluation](#2-n-gram-model-evaluation)
	- [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
	- [4. Vocabulary Analysis](#4-vocabulary-analysis)
	- [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
	- [6. Morphological Analysis (Experimental)](#6--morphological-analysis-experimental)
	- [7. Summary & Recommendations](#7-summary--recommendations)
	- [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
	- [Visualizations Index](#visualizations-index)

	---
	## 1. Tokenizer Evaluation

	![Tokenizer Compression](visualizations/tokenizer_compression.png)

	![Tokenizer Fertility](visualizations/tokenizer_fertility.png)

	![Tokenizer OOV](visualizations/tokenizer_oov.png)

	![Total Tokens](visualizations/tokenizer_total_tokens.png)

	### Results

	\| Vocab Size \| Compression \| Avg Token Len \| UNK Rate \| Total Tokens \|
	\|------------\|-------------\|---------------\|----------\|--------------\|
	\| 8k \| 4.501x \| 4.51 \| 0.2384% \| 99,847 \|
	\| 16k \| 4.662x \| 4.67 \| 0.2469% \| 96,404 \|
	\| 32k \| 4.818x \| 4.83 \| 0.2551% \| 93,284 \|
	\| 64k \| 4.935x 🏆 \| 4.95 \| 0.2614% \| 91,058 \|

	### Tokenization Examples

	Below are sample sentences tokenized with each vocabulary size:

	Sample 1: `ইথাক বিষ্ণুপ্রিয়া মণিপুরী ঠারর অনিয়মিত পত্রিকা আহান, যেহান সংগ্রাম সিংহর সম্পা...`

	\| Vocab \| Tokens \| Count \|
	\|-------\|--------\|-------\|
	\| 8k \| `▁ই থ াক ▁বিষ্ণুপ্রিয়া ▁মণিপুরী ▁ঠারর ▁অ নি য় মি ... (+21 more)` \| 31 \|
	\| 16k \| `▁ই থ াক ▁বিষ্ণুপ্রিয়া ▁মণিপুরী ▁ঠারর ▁অ নি য় মিত ... (+18 more)` \| 28 \|
	\| 32k \| `▁ই থ াক ▁বিষ্ণুপ্রিয়া ▁মণিপুরী ▁ঠারর ▁অনি য়মিত ▁পত্রিকা ▁আহান ... (+13 more)` \| 23 \|
	\| 64k \| `▁ইথাক ▁বিষ্ণুপ্রিয়া ▁মণিপুরী ▁ঠারর ▁অনিয়মিত ▁পত্রিকা ▁আহান , ▁যেহান ▁সংগ্রাম ... (+8 more)` \| 18 \|

	Sample 2: `.এমও(.mo) এগ মাকাউর নাঙে লেপকরিসি চিঙপা ডমেইনগ (ccTLD)। মিলাপ আইএএনএ-র মাকাউর তথ...`

	\| Vocab \| Tokens \| Count \|
	\|-------\|--------\|-------\|
	\| 8k \| `▁. এম ও (. mo ) ▁এগ ▁মাকা উর ▁নাঙে ... (+23 more)` \| 33 \|
	\| 16k \| `▁. এম ও (. mo ) ▁এগ ▁মাকা উর ▁নাঙে ... (+23 more)` \| 33 \|
	\| 32k \| `▁. এম ও (. mo ) ▁এগ ▁মাকাউর ▁নাঙে ▁লেপকরিসি ... (+21 more)` \| 31 \|
	\| 64k \| `▁. এম ও (. mo ) ▁এগ ▁মাকাউর ▁নাঙে ▁লেপকরিসি ... (+21 more)` \| 31 \|

	Sample 3: `বাংলাদেশর স্থানীয় সরকারর সিজিলে আসেতাই জিলা পরিষদ সিটি কর্পোরেশন (৬গ) থানা বারো...`

	\| Vocab \| Tokens \| Count \|
	\|-------\|--------\|-------\|
	\| 8k \| `▁বাংলাদেশর ▁স্ থান ীয় ▁সরকারর ▁সিজিল ে ▁আসেতাই ▁জিলা ▁পরিষ ... (+21 more)` \| 31 \|
	\| 16k \| `▁বাংলাদেশর ▁স্থানীয় ▁সরকারর ▁সিজিল ে ▁আসেতাই ▁জিলা ▁পরিষদ ▁সিটি ▁কর্পোরেশন ... (+15 more)` \| 25 \|
	\| 32k \| `▁বাংলাদেশর ▁স্থানীয় ▁সরকারর ▁সিজিল ে ▁আসেতাই ▁জিলা ▁পরিষদ ▁সিটি ▁কর্পোরেশন ... (+15 more)` \| 25 \|
	\| 64k \| `▁বাংলাদেশর ▁স্থানীয় ▁সরকারর ▁সিজিল ে ▁আসেতাই ▁জিলা ▁পরিষদ ▁সিটি ▁কর্পোরেশন ... (+15 more)` \| 25 \|


	### Key Findings

	- Best Compression: 64k achieves 4.935x compression
	- Lowest UNK Rate: 8k with 0.2384% unknown tokens
	- Trade-off: Larger vocabularies improve compression but increase model size
	- Recommendation: 32k vocabulary provides optimal balance for production use

	---
	## 2. N-gram Model Evaluation

	![N-gram Perplexity](visualizations/ngram_perplexity.png)

	![N-gram Unique](visualizations/ngram_unique.png)

	![N-gram Coverage](visualizations/ngram_coverage.png)

	### Results

	\| N-gram \| Variant \| Perplexity \| Entropy \| Unique N-grams \| Top-100 Coverage \| Top-1000 Coverage \|
	\|--------\|---------\|------------\|---------\|----------------\|------------------\|-------------------\|
	\| 2-gram \| Word \| 917 \| 9.84 \| 15,091 \| 44.2% \| 86.3% \|
	\| 2-gram \| Subword \| 598 🏆 \| 9.22 \| 14,901 \| 51.1% \| 92.9% \|
	\| 3-gram \| Word \| 1,565 \| 10.61 \| 31,633 \| 38.0% \| 79.5% \|
	\| 3-gram \| Subword \| 1,912 \| 10.90 \| 68,690 \| 32.6% \| 79.7% \|
	\| 4-gram \| Word \| 2,617 \| 11.35 \| 60,965 \| 35.0% \| 72.0% \|
	\| 4-gram \| Subword \| 3,535 \| 11.79 \| 166,549 \| 26.1% \| 72.8% \|
	\| 5-gram \| Word \| 3,304 \| 11.69 \| 65,705 \| 33.6% \| 68.3% \|
	\| 5-gram \| Subword \| 4,752 \| 12.21 \| 229,112 \| 22.8% \| 68.8% \|

	### Top 5 N-grams by Size

	2-grams (Word):

	\| Rank \| N-gram \| Count \|
	\|------\|--------\|-------\|
	\| 1 \| `সাক্ষরতার হারহান` \| 26,823 \|
	\| 2 \| `অতার মা` \| 20,497 \|
	\| 3 \| `জনসংখ্যার উপাত্ত` \| 19,704 \|
	\| 4 \| `জনসংখ্যা ইলাতাই` \| 19,552 \|
	\| 5 \| `লোক গননা` \| 19,533 \|

	3-grams (Word):

	\| Rank \| N-gram \| Count \|
	\|------\|--------\|-------\|
	\| 1 \| `মানুলেহা লোক গননা` \| 19,527 \|
	\| 2 \| `মারির মানুলেহা লোক` \| 19,526 \|
	\| 3 \| `অতার মা মুনি` \| 16,569 \|
	\| 4 \| `গ অতার মা` \| 15,694 \|
	\| 5 \| `লোক গননা অনুসারে` \| 14,182 \|

	4-grams (Word):

	\| Rank \| N-gram \| Count \|
	\|------\|--------\|-------\|
	\| 1 \| `মারির মানুলেহা লোক গননা` \| 19,525 \|
	\| 2 \| `গ অতার মা মুনি` \| 15,620 \|
	\| 3 \| `মানুলেহা লোক গননা অনুসারে` \| 14,181 \|
	\| 4 \| `অক্ষাংশ বারো দ্রাঘিমাংশ ইলতাই` \| 9,366 \|
	\| 5 \| `মাপাহানর অক্ষাংশ বারো দ্রাঘিমাংশ` \| 9,315 \|

	5-grams (Word):

	\| Rank \| N-gram \| Count \|
	\|------\|--------\|-------\|
	\| 1 \| `মারির মানুলেহা লোক গননা অনুসারে` \| 14,180 \|
	\| 2 \| `মাপাহানর অক্ষাংশ বারো দ্রাঘিমাংশ ইলতাই` \| 9,315 \|
	\| 3 \| `এহার মাপাহানর অক্ষাংশ বারো দ্রাঘিমাংশ` \| 9,310 \|
	\| 4 \| `এহানর গড় উচ হান ইলতাই` \| 6,096 \|
	\| 5 \| `মান্নাহাত্ত এহানর গড় উচ হান` \| 6,096 \|

	2-grams (Subword):

	\| Rank \| N-gram \| Count \|
	\|------\|--------\|-------\|
	\| 1 \| `র _` \| 407,202 \|
	\| 2 \| `। _` \| 163,086 \|
	\| 3 \| `হা ন` \| 154,676 \|
	\| 4 \| `ন _` \| 147,838 \|
	\| 5 \| `_ মা` \| 138,460 \|

	3-grams (Subword):

	\| Rank \| N-gram \| Count \|
	\|------\|--------\|-------\|
	\| 1 \| `র _ মা` \| 95,254 \|
	\| 2 \| `হা ন _` \| 94,536 \|
	\| 3 \| `_ বা রো` \| 68,915 \|
	\| 4 \| `বা রো _` \| 68,891 \|
	\| 5 \| `_ ই উ` \| 64,643 \|

	4-grams (Subword):

	\| Rank \| N-gram \| Count \|
	\|------\|--------\|-------\|
	\| 1 \| `_ বা রো _` \| 68,886 \|
	\| 2 \| `_ ই উ নি` \| 64,359 \|
	\| 3 \| `ই উ নি য়` \| 55,648 \|
	\| 4 \| `উ নি য় ন` \| 55,615 \|
	\| 5 \| `জ ন সং খ্যা` \| 44,873 \|

	5-grams (Subword):

	\| Rank \| N-gram \| Count \|
	\|------\|--------\|-------\|
	\| 1 \| `_ ই উ নি য়` \| 55,620 \|
	\| 2 \| `ই উ নি য় ন` \| 55,614 \|
	\| 3 \| `_ জ ন সং খ্যা` \| 44,868 \|
	\| 4 \| `_ উ পা ত্ত _` \| 36,516 \|
	\| 5 \| `_ পৌ র স ভা` \| 34,339 \|


	### Key Findings

	- Best Perplexity: 2-gram (subword) with 598
	- Entropy Trend: Decreases with larger n-grams (more predictable)
	- Coverage: Top-1000 patterns cover ~69% of corpus
	- Recommendation: 4-gram or 5-gram for best predictive performance

	---
	## 3. Markov Chain Evaluation

	![Markov Entropy](visualizations/markov_entropy.png)

	![Markov Contexts](visualizations/markov_contexts.png)

	![Markov Branching](visualizations/markov_branching.png)

	### Results

	\| Context \| Variant \| Avg Entropy \| Perplexity \| Branching Factor \| Unique Contexts \| Predictability \|
	\|---------\|---------\|-------------\|------------\|------------------\|-----------------\|----------------\|
	\| 1 \| Word \| 0.7841 \| 1.722 \| 4.39 \| 60,191 \| 21.6% \|
	\| 1 \| Subword \| 1.0505 \| 2.071 \| 11.75 \| 3,037 \| 0.0% \|
	\| 2 \| Word \| 0.1820 \| 1.134 \| 1.54 \| 262,172 \| 81.8% \|
	\| 2 \| Subword \| 0.6365 \| 1.555 \| 3.68 \| 35,639 \| 36.4% \|
	\| 3 \| Word \| 0.0756 \| 1.054 \| 1.27 \| 399,673 \| 92.4% \|
	\| 3 \| Subword \| 0.4888 \| 1.403 \| 2.43 \| 130,940 \| 51.1% \|
	\| 4 \| Word \| 0.0494 🏆 \| 1.035 \| 1.19 \| 504,719 \| 95.1% \|
	\| 4 \| Subword \| 0.3613 \| 1.285 \| 1.77 \| 317,931 \| 63.9% \|

	### Generated Text Samples (Word-based)

	Below are text samples generated from each word-based Markov chain model:

	Context Size 1:

	1. `বারো জিলা বেয়াপা ১৫ ৪৪ ৮২৮ মিটার ফুট জনসংখ্যার উপাত্ত পৌরসভা আহান ভৌগলিক উপাত্ত ব্রাজিলর ঔয়াংমুঙ`
	2. `ইউনিয়ন এগত গাঙ বারো ফুংগালাইরু বুলিয়া কিত্তাও নেই অহাত্তবারো এহার আয়তন লয়াহান ৪১৬ গ অতার মা`
	3. `উপাত্ত শহর এহার আয়তন লয়াহান ৩৫৪ গ অতার মা হারহান ৫৯ ৫ অহানাত্ত এস নইচত জনসংখ্যার`

	Context Size 2:

	1. `সাক্ষরতার হারহান ৫৯ ৫ অহানাত্ত গঞ্জাম এহানর সাক্ষরতার হারহান ৭২ মুনির মা সাক্ষরতার হারহান ৬৫ বারো হু...`
	2. `অতার মা মুনি ৫০ বারো জিলা বেয়াপা এরে পৌরসভার মানু শহরেদে বারো ১১ ৭৩৬গ গাঙেদে থাইতারা হারি`
	3. `জনসংখ্যার উপাত্ত ভারতর মারির মানুলেহা লোক গননা অনুসারে আলসটের কাউন্টি ইংরেজি oglethorpe county এহান ...`

	Context Size 3:

	1. `মানুলেহা লোক গননা অনুসারে বার্বোসা পৌরসভাহানর জনসংখ্যা ইলাতাই ১০ ৪২৫ গ অতার মা মুনি ৫০ বারো জিলা বেয...`
	2. `মারির মানুলেহা লোক গননা অনুসারে পালেসটিনা ডে গোয়াস পর্তুগীজ santa bárbara de goiás এহান ব্রাজিলর হম...`
	3. `অতার মা মুনি ৫১ বারো জেলা বেয়াপা ৪৯ এহানাত সাক্ষরতার হারহান ৭৩ বারো জেলার মা হারহান ৬৮ আস্তা`

	Context Size 4:

	1. `মারির মানুলেহা লোক গননা অনুসারে টের্রেবোন পারিশ র জনসংখ্যা ইলাতাই ৮৭ ৯০৪ গ ৩২ ৭৩২গ ঘরর ইউনিট আসে হার...`
	2. `গ অতার মা মুনি ৫২ বারো জিলা বেয়াপা এরে পৌরসভার মানু ৪২৩গ শহরেদে বারো গাঙেদে থাইতারা হারি বর্গ কিলোম...`
	3. `মানুলেহা লোক গননা অনুসারে ক্লাবেরাস কাউন্টি র জনসংখ্যা ইলাতাই ১৮ ৫৬১ গ ঘরর ইউনিট আসে চৌদ্দাহান মুঙেদ...`


	### Generated Text Samples (Subword-based)

	Below are text samples generated from each subword-based Markov chain model:

	Context Size 1:

	1. `_মারিসিতার_ঔয়াঙেদে:_মুনিয়`
	2. `রসভা_সাক্ষর_শহর_পৌর_ই`
	3. `নর_অক্টোবসভার_হানিয়ন।_`

	Context Size 2:

	1. `র_সাই_৬৬%।_ঔয়াঙেদে_থা_`
	2. `।_অনুসারে_৩১তম_বিয়া_জিলা`
	3. `হান_এহান_ইউনিয়নর_সান্টা_`

	Context Size 3:

	1. `র_মা_সাক্ষরতার_হারহান_৫৯.`
	2. `হান_৭৯%,_অতার_হারহান_(`
	3. `_বারো_গাঙেদে_থাইতারা।_হারি_ব`

	Context Size 4:

	1. `_বারো_জিলা/বেয়াপা_(১৫-৪৪_ব`
	2. `_ইউনিট_আসে।_চৌদ্দাহান_মুঙেদে:`
	3. `ইউনিয়ন_আগ।_ভৌগলিক_উপাত্ত_`


	### Key Findings

	- Best Predictability: Context-4 (word) with 95.1% predictability
	- Branching Factor: Decreases with context size (more deterministic)
	- Memory Trade-off: Larger contexts require more storage (317,931 contexts)
	- Recommendation: Context-3 or Context-4 for text generation

	---
	## 4. Vocabulary Analysis

	![Zipf's Law](visualizations/zipf_law.png)

	![Top Words](visualizations/top20_words.png)

	![Coverage Curve](visualizations/vocab_coverage.png)

	### Statistics

	\| Metric \| Value \|
	\|--------\|-------\|
	\| Vocabulary Size \| 32,965 \|
	\| Total Tokens \| 2,030,616 \|
	\| Mean Frequency \| 61.60 \|
	\| Median Frequency \| 3 \|
	\| Frequency Std Dev \| 897.18 \|

	### Most Common Words

	\| Rank \| Word \| Frequency \|
	\|------\|------\|-----------\|
	\| 1 \| বারো \| 68,888 \|
	\| 2 \| ইউনিয়ন \| 42,535 \|
	\| 3 \| উপাত্ত \| 36,516 \|
	\| 4 \| হারহান \| 31,910 \|
	\| 5 \| মা \| 31,022 \|
	\| 6 \| মানু \| 30,460 \|
	\| 7 \| সাক্ষরতার \| 26,839 \|
	\| 8 \| গ \| 26,421 \|
	\| 9 \| অতার \| 25,584 \|
	\| 10 \| জনসংখ্যার \| 24,823 \|

	### Least Common Words (from vocabulary)

	\| Rank \| Word \| Frequency \|
	\|------\|------\|-----------\|
	\| 1 \| সুখর \| 2 \|
	\| 2 \| পরিত্যাগ \| 2 \|
	\| 3 \| মালতী \| 2 \|
	\| 4 \| আকগও \| 2 \|
	\| 5 \| ক্ষনিক \| 2 \|
	\| 6 \| সযন্তে \| 2 \|
	\| 7 \| কণ্টক \| 2 \|
	\| 8 \| পরিহার \| 2 \|
	\| 9 \| বিরোধিতা \| 2 \|
	\| 10 \| অপরাপর \| 2 \|

	### Zipf's Law Analysis

	\| Metric \| Value \|
	\|--------\|-------\|
	\| Zipf Coefficient \| 1.3137 \|
	\| R² (Goodness of Fit) \| 0.980288 \|
	\| Adherence Quality \| excellent \|

	### Coverage Analysis

	\| Top N Words \| Coverage \|
	\|-------------\|----------\|
	\| Top 100 \| 62.6% \|
	\| Top 1,000 \| 89.9% \|
	\| Top 5,000 \| 95.0% \|
	\| Top 10,000 \| 96.8% \|

	### Key Findings

	- Zipf Compliance: R²=0.9803 indicates excellent adherence to Zipf's law
	- High Frequency Dominance: Top 100 words cover 62.6% of corpus
	- Long Tail: 22,965 words needed for remaining 3.2% coverage

	---
	## 5. Word Embeddings Evaluation

	![Embedding Isotropy](visualizations/embedding_isotropy.png)

	![Similarity Matrix](visualizations/embedding_similarity.png)

	![t-SNE Words](visualizations/tsne_words.png)

	![t-SNE Sentences](visualizations/tsne_sentences.png)


	### 5.1 Cross-Lingual Alignment

	![Alignment Quality](visualizations/embedding_alignment_quality.png)

	![Multilingual t-SNE](visualizations/embedding_tsne_multilingual.png)


	### 5.2 Model Comparison

	\| Model \| Dimension \| Isotropy \| Semantic Density \| Alignment R@1 \| Alignment R@10 \|
	\|-------\|-----------\|----------\|------------------\|---------------\|----------------\|
	\| mono_32d \| 32 \| 0.6926 🏆 \| 0.3671 \| N/A \| N/A \|
	\| mono_64d \| 64 \| 0.5161 \| 0.3444 \| N/A \| N/A \|
	\| mono_128d \| 128 \| 0.2440 \| 0.3266 \| N/A \| N/A \|
	\| aligned_32d \| 32 \| 0.6926 \| 0.3703 \| 0.0100 \| 0.0740 \|
	\| aligned_64d \| 64 \| 0.5161 \| 0.3426 \| 0.0240 \| 0.1200 \|
	\| aligned_128d \| 128 \| 0.2440 \| 0.3276 \| 0.0380 \| 0.1340 \|

	### Key Findings

	- Best Isotropy: mono_32d with 0.6926 (more uniform distribution)
	- Semantic Density: Average pairwise similarity of 0.3465. Lower values indicate better semantic separation.
	- Alignment Quality: Aligned models achieve up to 3.8% R@1 in cross-lingual retrieval.
	- Recommendation: 128d aligned for best cross-lingual performance

	---
	## 6. Morphological Analysis (Experimental)

	This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.

	### 6.1 Productivity & Complexity

	\| Metric \| Value \| Interpretation \| Recommendation \|
	\|--------\|-------\|----------------\|----------------\|
	\| Productivity Index \| 5.000 \| High morphological productivity \| Reliable analysis \|
	\| Idiomaticity Gap \| -0.006 \| Low formulaic content \| - \|

	### 6.2 Affix Inventory (Productive Units)

	These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.

	#### Productive Prefixes
	\| Prefix \| Examples \|
	\|--------\|----------\|
	\| `-কা` \| কানেডো, কাইতলী, কানিনা \|

	#### Productive Suffixes
	\| Suffix \| Examples \|
	\|--------\|----------\|
	\| `-া` \| বারুইয়া, খানা, বুললা \|
	\| `-র` \| ০০০র, চাটমোহর, ফুর \|
	\| `-়া` \| বারুইয়া, বেলেয়া, ভরাপাড়া \|
	\| `-য়া` \| বারুইয়া, বেলেয়া, বড়হাতিয়া \|
	\| `-ুর` \| ফুর, গোপালপুর, সরদারপুর \|
	\| `-পুর` \| গোপালপুর, সরদারপুর, কুতবউল্লাপুর \|
	\| `-িয়া` \| বড়হাতিয়া, বাসুন্দিয়া, ঘাটলোদিয়া \|
	\| `-রা` \| ভাদ্রা, ভাটারা, মোরেইরা \|

	### 6.3 Bound Stems (Lexical Roots)

	Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.

	No significant bound stems detected.


	### 6.4 Affix Compatibility (Co-occurrence)

	This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.

	\| Prefix \| Suffix \| Frequency \| Examples \|
	\|--------\|--------\|-----------\|----------\|
	\| `-কা` \| `-া` \| 44 words \| কারোবা, কাটাৱাবা \|
	\| `-কা` \| `-র` \| 41 words \| কামর, কান্নানুর \|
	\| `-কা` \| `-ুর` \| 15 words \| কান্নানুর, কাজীপুর \|
	\| `-কা` \| `-়া` \| 15 words \| কাদিরপাড়া, কালকরিয়া \|
	\| `-কা` \| `-য়া` \| 10 words \| কালকরিয়া, কালাবাড়িয়া \|
	\| `-কা` \| `-িয়া` \| 10 words \| কালকরিয়া, কালাবাড়িয়া \|
	\| `-কা` \| `-পুর` \| 5 words \| কাজীপুর, কালিদাসপুর \|
	\| `-কা` \| `-রা` \| 5 words \| কাংরা, কাকৈরগরা \|

	### 6.5 Recursive Morpheme Segmentation

	Using Recursive Hierarchical Substitutability, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).

	\| Word \| Suggested Split \| Confidence \| Stem \|
	\|------\|-----------------\|------------\|------\|
	\| জাঙ্গালিয়া \| `জাঙ্গাল-িয়া` \| 4.5 \| `জাঙ্গাল` \|
	\| মাখদুমপুর \| `মাখদুম-পুর` \| 4.5 \| `মাখদুম` \|
	\| স্লোভাকিয়া \| `স্লোভাক-িয়া` \| 4.5 \| `স্লোভাক` \|
	\| বাল্লাপুর \| `বাল্লা-পুর` \| 4.5 \| `বাল্লা` \|
	\| ওসমানীয়া \| `ওসমানী-য়া` \| 4.5 \| `ওসমানী` \|
	\| কাসকালহেইরা \| `কা-সকালহেই-রা` \| 3.0 \| `সকালহেই` \|
	\| কারুপ্পুর \| `কা-রুপ্-পুর` \| 3.0 \| `রুপ্` \|
	\| বাহাদুরপুর \| `বাহাদ-ুর-পুর` \| 3.0 \| `বাহাদ` \|
	\| কাফেলান্ডিয়া \| `কা-ফেলান্ড-িয়া` \| 3.0 \| `ফেলান্ড` \|
	\| ইটাকোয়াটিয়ারা \| `ইটাকোয়াট-িয়া-রা` \| 3.0 \| `ইটাকোয়াট` \|
	\| পীরযাত্রাপুর \| `পীরযাত্-রা-পুর` \| 3.0 \| `পীরযাত্` \|
	\| কাসসিলান্ডিয়া \| `কা-সসিলান্ড-িয়া` \| 3.0 \| `সসিলান্ড` \|
	\| কাশালিয়া \| `কা-শালি-য়া` \| 3.0 \| `শালি` \|
	\| কাউন্দিয়া \| `কা-উন্দ-িয়া` \| 3.0 \| `উন্দ` \|
	\| কান্নানুর \| `কা-ন্নান-ুর` \| 3.0 \| `ন্নান` \|

	### 6.6 Linguistic Interpretation

	> Automated Insight:
	The language Bishnupriya shows high morphological productivity. The subword models are significantly more efficient than word models, suggesting a rich system of affixation or compounding.

	---
	## 7. Summary & Recommendations

	![Performance Dashboard](visualizations/performance_dashboard.png)

	### Production Recommendations

	\| Component \| Recommended \| Rationale \|
	\|-----------\|-------------\|-----------\|
	\| Tokenizer \| 64k BPE \| Best compression (4.94x) \|
	\| N-gram \| 2-gram \| Lowest perplexity (598) \|
	\| Markov \| Context-4 \| Highest predictability (95.1%) \|
	\| Embeddings \| 100d \| Balanced semantic capture and isotropy \|


	---
	## Appendix: Metrics Glossary & Interpretation Guide

	This section provides definitions, intuitions, and guidance for interpreting the metrics used throughout this report.

	### Tokenizer Metrics

	Compression Ratio
	> Definition: The ratio of characters to tokens (chars/token). Measures how efficiently the tokenizer represents text.
	>
	> Intuition: Higher compression means fewer tokens needed to represent the same text, reducing sequence lengths for downstream models. A 3x compression means ~3 characters per token on average.
	>
	> What to seek: Higher is generally better for efficiency, but extremely high compression may indicate overly aggressive merging that loses morphological information.

	Average Token Length (Fertility)
	> Definition: Mean number of characters per token produced by the tokenizer.
	>
	> Intuition: Reflects the granularity of tokenization. Longer tokens capture more context but may struggle with rare words; shorter tokens are more flexible but increase sequence length.
	>
	> What to seek: Balance between 2-5 characters for most languages. Arabic/morphologically-rich languages may benefit from slightly longer tokens.

	Unknown Token Rate (OOV Rate)
	> Definition: Percentage of tokens that map to the unknown/UNK token, indicating words the tokenizer cannot represent.
	>
	> Intuition: Lower OOV means better vocabulary coverage. High OOV indicates the tokenizer encounters many unseen character sequences.
	>
	> What to seek: Below 1% is excellent; below 5% is acceptable. BPE tokenizers typically achieve very low OOV due to subword fallback.

	### N-gram Model Metrics

	Perplexity
	> Definition: Measures how "surprised" the model is by test data. Mathematically: 2^(cross-entropy). Lower values indicate better prediction.
	>
	> Intuition: If perplexity is 100, the model is as uncertain as if choosing uniformly among 100 options at each step. A perplexity of 10 means effectively choosing among 10 equally likely options.
	>
	> What to seek: Lower is better. Perplexity decreases with larger n-grams (more context). Values vary widely by language and corpus size.

	Entropy
	> Definition: Average information content (in bits) needed to encode the next token given the context. Related to perplexity: perplexity = 2^entropy.
	>
	> Intuition: High entropy means high uncertainty/randomness; low entropy means predictable patterns. Natural language typically has entropy between 1-4 bits per character.
	>
	> What to seek: Lower entropy indicates more predictable text patterns. Entropy should decrease as n-gram size increases.

	Coverage (Top-K)
	> Definition: Percentage of corpus occurrences explained by the top K most frequent n-grams.
	>
	> Intuition: High coverage with few patterns indicates repetitive/formulaic text; low coverage suggests diverse vocabulary usage.
	>
	> What to seek: Depends on use case. For language modeling, moderate coverage (40-60% with top-1000) is typical for natural text.

	### Markov Chain Metrics

	Average Entropy
	> Definition: Mean entropy across all contexts, measuring average uncertainty in next-word prediction.
	>
	> Intuition: Lower entropy means the model is more confident about what comes next. Context-1 has high entropy (many possible next words); Context-4 has low entropy (few likely continuations).
	>
	> What to seek: Decreasing entropy with larger context sizes. Very low entropy (<0.1) indicates highly deterministic transitions.

	Branching Factor
	> Definition: Average number of unique next tokens observed for each context.
	>
	> Intuition: High branching = many possible continuations (flexible but uncertain); low branching = few options (predictable but potentially repetitive).
	>
	> What to seek: Branching factor should decrease with context size. Values near 1.0 indicate nearly deterministic chains.

	Predictability
	> Definition: Derived metric: (1 - normalized_entropy) × 100%. Indicates how deterministic the model's predictions are.
	>
	> Intuition: 100% predictability means the next word is always certain; 0% means completely random. Real text falls between these extremes.
	>
	> What to seek: Higher predictability for text generation quality, but too high (>98%) may produce repetitive output.

	### Vocabulary & Zipf's Law Metrics

	Zipf's Coefficient
	> Definition: The slope of the log-log plot of word frequency vs. rank. Zipf's law predicts this should be approximately -1.
	>
	> Intuition: A coefficient near -1 indicates the corpus follows natural language patterns where a few words are very common and most words are rare.
	>
	> What to seek: Values between -0.8 and -1.2 indicate healthy natural language distribution. Deviations may suggest domain-specific or artificial text.

	R² (Coefficient of Determination)
	> Definition: Measures how well the linear fit explains the frequency-rank relationship. Ranges from 0 to 1.
	>
	> Intuition: R² near 1.0 means the data closely follows Zipf's law; lower values indicate deviation from expected word frequency patterns.
	>
	> What to seek: R² > 0.95 is excellent; > 0.99 indicates near-perfect Zipf adherence typical of large natural corpora.

	Vocabulary Coverage
	> Definition: Cumulative percentage of corpus tokens accounted for by the top N words.
	>
	> Intuition: Shows how concentrated word usage is. If top-100 words cover 50% of text, the corpus relies heavily on common words.
	>
	> What to seek: Top-100 covering 30-50% is typical. Higher coverage indicates more repetitive text; lower suggests richer vocabulary.

	### Word Embedding Metrics

	Isotropy
	> Definition: Measures how uniformly distributed vectors are in the embedding space. Computed as the ratio of minimum to maximum singular values.
	>
	> Intuition: High isotropy (near 1.0) means vectors spread evenly in all directions; low isotropy means vectors cluster in certain directions, reducing expressiveness.
	>
	> What to seek: Higher isotropy generally indicates better-quality embeddings. Values > 0.1 are reasonable; > 0.3 is good. Lower-dimensional embeddings tend to have higher isotropy.

	Average Norm
	> Definition: Mean magnitude (L2 norm) of word vectors in the embedding space.
	>
	> Intuition: Indicates the typical "length" of vectors. Consistent norms suggest stable training; high variance may indicate some words are undertrained.
	>
	> What to seek: Relatively consistent norms across models. The absolute value matters less than consistency (low std deviation).

	Cosine Similarity
	> Definition: Measures angular similarity between vectors, ranging from -1 (opposite) to 1 (identical direction).
	>
	> Intuition: Words with similar meanings should have high cosine similarity. This is the standard metric for semantic relatedness in embeddings.
	>
	> What to seek: Semantically related words should score > 0.5; unrelated words should be near 0. Synonyms often score > 0.7.

	t-SNE Visualization
	> Definition: t-Distributed Stochastic Neighbor Embedding - a dimensionality reduction technique that preserves local structure for visualization.
	>
	> Intuition: Clusters in t-SNE plots indicate groups of semantically related words. Spread indicates vocabulary diversity; tight clusters suggest semantic coherence.
	>
	> What to seek: Meaningful clusters (e.g., numbers together, verbs together). Avoid over-interpreting distances - t-SNE preserves local, not global, structure.

	### General Interpretation Guidelines

	1. Compare within model families: Metrics are most meaningful when comparing models of the same type (e.g., 8k vs 64k tokenizer).
	2. Consider trade-offs: Better performance on one metric often comes at the cost of another (e.g., compression vs. OOV rate).
	3. Context matters: Optimal values depend on downstream tasks. Text generation may prioritize different metrics than classification.
	4. Corpus influence: All metrics are influenced by corpus characteristics. Wikipedia text differs from social media or literature.
	5. Language-specific patterns: Morphologically rich languages (like Arabic) may show different optimal ranges than analytic languages.


	### Visualizations Index

	\| Visualization \| Description \|
	\|---------------\|-------------\|
	\| Tokenizer Compression \| Compression ratios by vocabulary size \|
	\| Tokenizer Fertility \| Average token length by vocabulary \|
	\| Tokenizer OOV \| Unknown token rates \|
	\| Tokenizer Total Tokens \| Total tokens by vocabulary \|
	\| N-gram Perplexity \| Perplexity by n-gram size \|
	\| N-gram Entropy \| Entropy by n-gram size \|
	\| N-gram Coverage \| Top pattern coverage \|
	\| N-gram Unique \| Unique n-gram counts \|
	\| Markov Entropy \| Entropy by context size \|
	\| Markov Branching \| Branching factor by context \|
	\| Markov Contexts \| Unique context counts \|
	\| Zipf's Law \| Frequency-rank distribution with fit \|
	\| Vocab Frequency \| Word frequency distribution \|
	\| Top 20 Words \| Most frequent words \|
	\| Vocab Coverage \| Cumulative coverage curve \|
	\| Embedding Isotropy \| Vector space uniformity \|
	\| Embedding Norms \| Vector magnitude distribution \|
	\| Embedding Similarity \| Word similarity heatmap \|
	\| Nearest Neighbors \| Similar words for key terms \|
	\| t-SNE Words \| 2D word embedding visualization \|
	\| t-SNE Sentences \| 2D sentence embedding visualization \|
	\| Position Encoding \| Encoding method comparison \|
	\| Model Sizes \| Storage requirements \|
	\| Performance Dashboard \| Comprehensive performance overview \|

	---
	## About This Project

	### Data Source

	Models trained on [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly) - a monthly snapshot of Wikipedia articles across 300+ languages.

	### Project

	A project by [Wikilangs](https://wikilangs.org) - Open-source NLP models for every Wikipedia language.

	### Maintainer

	[Omar Kamali](https://omarkamali.com) - [Omneity Labs](https://omneitylabs.com)

	### Citation

	If you use these models in your research, please cite:

	```bibtex
	@misc{wikilangs2025,
	author = {Kamali, Omar},
	title = {Wikilangs: Open NLP Models for Wikipedia Languages},
	year = {2025},
	doi = {10.5281/zenodo.18073153},
	publisher = {Zenodo},
	url = {https://huggingface.co/wikilangs}
	institution = {Omneity Labs}
	}
	```

	### License

	MIT License - Free for academic and commercial use.

	### Links

	- 🌐 Website: [wikilangs.org](https://wikilangs.org)
	- 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
	- 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
	- 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
	- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
	---
	Generated by Wikilangs Models Pipeline

	Report Date: 2026-01-03 19:21:34