Update README.md

README.md

@@ -17,32 +17,24 @@ metrics:
 
 # CommonLingua
 
-Byte-level language identification model for 334 languages — 2.35 M parameters, 5-9 MB on disk, runs on CPU.
+CommonLingua is a 2.35 million-parameters language identification model trained on 2,482,568 paragraphs from Structured Wikipedia and Common Corpus](https://huggingface.co/datasets/PleIAs/common_corpus). As of 2026, CommonLingua is the best performing model on the CommonLID benchmark with significant gains over the previous SOTA.
 
-CommonLingua sorts raw web, PDF, and digitised text into 334 ISO 639-3 language buckets so it can feed downstream training pipelines. It was built and trained at [PleIAs](https://pleias.fr) for curating [Common Corpus](https://huggingface.co/datasets/PleIAs/common_corpus), with a particular focus on the long tail — 61 African languages are supported, including languages with almost no coverage.
+CommonLingua is based on a byte-level hybrid architecture combining three conv1D layers with an attention layer. It was originally designed for large scale classification of pretraining data and intently trained on diverse data sources, especially realistic documents with OCR errors as well as a a particular focus on the long tail — 61 African languages are supported, including languages with almost no coverage.
 
-CommonLingua is trained exclusively on open data under free licensed. We release the original dataset made of 2,482,568 paragraphs across 334 languages, drawn from Structured Wikipedia and other Common Corpus subsets.
-
-Per-source contributions, license attribution, and full schema are documented in [PleIAs/CommonLingua-Train](https://huggingface.co/datasets/PleIAs/CommonLingua-Train).
+Since CommonLingua is trained exclusively on open data under free license, we release the extent [original dataset](https://huggingface.co/datasets/PleIAs/CommonLingua-Train) with detailed licensing contribution.
 
 ## Architecture
 
-```
-raw bytes → [trigram hash embed (4096 × 64)]
-         ↓                                    ↘
-         + ────────→ 3× depthwise Conv1D (k=15) → 1× attention (RoPE, 4 heads)
-                                                         ↓
-                                               masked mean-pool → 334 logits
-```
+CommonLingua uses a new original architecture, optimized for task accuracy in an extremely small model size range. 
 
 - **No tokenizer.** The model operates directly on raw UTF-8 bytes, padded to 512. This makes it inherently script-agnostic — Latin, Arabic, Ethiopic, N'Ko, Tifinagh, Devanagari, CJK, all handled by the same byte stream.
-- **Trigram hash embedding** (added in v7.2.1): a polynomial rolling hash of byte 3-grams indexes a 4096-bucket embedding table. Hash collisions act as regularisation. Adds ~262 k parameters and ≤ 2% inference overhead, but improves macro F1 by +1.2 points and African F1 by +1.5 points over the no-n-gram baseline.
+- **Trigram hash embedding**: a polynomial rolling hash of byte 3-grams indexes a 4096-bucket embedding table. Hash collisions act as regularisation. Our ablations showed the added signal improved macro F1 by +1.2 points over the non-gram baseline.
 - **Causal Conv1D × 3** captures local byte patterns (script ranges, common digraphs, morpheme boundaries).
 - **Bidirectional attention × 1 with RoPE** captures global structure across the paragraph.
 
 ## Evaluation
 
-Evaluated on **CommonLID** (Ortiz Suárez et al. 2026): 376 k held-out paragraphs, 200+ languages. All baselines are re-evaluated through the same pipeline (`iso639-lang` normalisation, equivalence-class collapsing applied identically) for an apples-to-apples comparison.
+We evaluated CommonLingua on CommonLID (Ortiz Suárez et al. 2026): 376 k held-out paragraphs, 200+ languages. All baselines are re-evaluated through the same pipeline (`iso639-lang` normalisation, equivalence-class collapsing applied identically) for an apples-to-apples comparison.
 
 | Model                | Params | Labels | Strict acc | Equiv acc | Macro F1   |
 |----------------------|-------:|-------:|----------:|----------:|-----------:|
@@ -51,11 +43,11 @@ Evaluated on **CommonLID** (Ortiz Suárez et al. 2026): 376 k held-out paragraph
 | GlotLID v3           | ~600 M | **2 102**  | 57.69 %   | 71.26 %   | 0.6729 |
 | **CommonLingua v7.2.1** | 2.35 M | 334 | **77.63 %** | **82.92 %** | **0.7879** |
 
-CommonLingua reaches +11.5 macro F1 over the next best baseline.
+CommonLingua reaches +11.5 macro F1 over the next best baseline. We discarded Lingala from our evaluation since most samples from CommonLID turned out to belong to other close languages.
 
 ### Throughput
 
-Texts/sec (one paragraph = one text, ≤ 512 bytes input, padded). Real CommonLingua weights and the production code path:
+We evaluated CommonLingua in texts/sec (one paragraph = one text, ≤ 512 bytes input, padded).
 
 | Device | Setting | fp32 | bf16 | bf16 vs fp32 |
 |---|---|---:|---:|---:|
@@ -63,8 +55,8 @@ Texts/sec (one paragraph = one text, ≤ 512 bytes input, padded). Real CommonLi
 | H100 80GB (bs=1024)             |            | 10,892 | 26,130 | 2.4× |
 | H100 80GB (bs=256)              |            | 10,677 | 25,241 | 2.4× |
 | H100 80GB (bs=64)               | low-latency| 10,025 | 22,625 | 2.3× |
-| Sapphire Rapids CPU (8 threads) | bs=32      | _PENDING_ | _PENDING_ | _PENDING_ |
-| Sapphire Rapids CPU (1 thread)  | bs=32      | _PENDING_ | _PENDING_ | _PENDING_ |
+| Sapphire Rapids CPU (8 threads) | bs=32      | 183 | **553** | 3.0× |
+| Sapphire Rapids CPU (1 thread)  | bs=32      | 44 | **114** | 2.6×|
 
 ## Quick start