v7.2.1-ngram release: fp32 + bf16 weights, model.py, predict.py, model card

README.md

@@ -0,0 +1,160 @@
+---
+license: apache-2.0
+language:
+- multilingual
+tags:
+- language-identification
+- lid
+- byte-level
+- corpus-curation
+- african-languages
+library_name: pytorch
+pipeline_tag: text-classification
+metrics:
+- f1
+- accuracy
+---
+
+# CommonLingua
+
+**Byte-level language identification for 334 languages — 2.35 M parameters, 9 MB on disk, runs on CPU.**
+
+CommonLingua sorts raw web, PDF, and digitised text into 334 ISO 639-3 language buckets so it can feed downstream training pipelines. It was built and trained at [PleIAs](https://pleias.fr) for curating [Common Corpus](https://huggingface.co/datasets/PleIAs/common_corpus), with a particular focus on the long tail — **61 African languages** are supported, including languages with no fastText / OpenLID coverage.
+
+|             |       |
+|---|---|
+| **Languages**           | 334 (61 African) |
+| **Parameters**          | 2,347,854 |
+| **Disk (fp32 / bf16)**  | 9.4 MB / 4.7 MB |
+| **Max input**           | 512 bytes (~paragraph) |
+| **CommonLID macro F1**  | **0.7879** |
+| **CommonLID strict acc**| 77.63% |
+| **License**             | Apache-2.0 |
+
+## Quick start
+
+```bash
+pip install "git+https://github.com/PleIAs/bytehybrid-lid#egg=commonlingua[hub]"
+```
+
+```python
+from commonlingua import LID
+
+lid = LID.from_pretrained("PleIAs/CommonLingua")          # auto-downloads
+# Use the bf16 build for 2× speedup on GPU at no measurable quality cost:
+# lid = LID.from_pretrained("PleIAs/CommonLingua", dtype="bf16")
+
+text = (
+    "Wikipédia est une encyclopédie universelle, multilingue, créée par Jimmy "
+    "Wales et Larry Sanger le 15 janvier 2001 et fonctionnant selon le principe "
+    "du wiki."
+)
+r = lid.predict(text)
+print(r.lang, r.confidence)   # fra 0.99
+```
+
+The intended workload is **paragraph-level corpus curation**. For batch annotation of large parquet shards, see `predict_parquet` in the package README.
+
+## Architecture
+
+```
+raw bytes → [trigram hash embed (4096 × 64)]
+         ↓                                    ↘
+         + ────────→ 3× depthwise Conv1D (k=15) → 1× attention (RoPE, 4 heads)
+                                                         ↓
+                                               masked mean-pool → 334 logits
+```
+
+- **No tokenizer.** The model operates directly on raw UTF-8 bytes, padded to 512. This makes it inherently script-agnostic — Latin, Arabic, Ethiopic, N'Ko, Tifinagh, Devanagari, CJK, all handled by the same byte stream.
+- **Trigram hash embedding** (added in v7.2.1): a polynomial rolling hash of byte 3-grams indexes a 4096-bucket embedding table. Hash collisions act as regularisation. Adds ~262 k parameters and ≤ 2% inference overhead, but improves macro F1 by +1.2 points and African F1 by +1.5 points over the no-n-gram baseline.
+- **Causal Conv1D × 3** captures local byte patterns (script ranges, common digraphs, morpheme boundaries).
+- **Bidirectional attention × 1 with RoPE** captures global structure across the paragraph.
+
+## Evaluation
+
+Evaluated on **CommonLID** (Ortiz Suárez et al. 2026): 376 k held-out paragraphs, 200+ languages. All baselines re-evaluated through the same pipeline (`iso639-lang` normalisation, equivalence-class collapsing applied identically) for an apples-to-apples comparison.
+
+### Headline
+
+| Model                | Params | Labels | Strict acc | Equiv acc | Macro F1   |
+|----------------------|-------:|-------:|----------:|----------:|-----------:|
+| OpenLID v2           | ~600 M |   200  | 55.77 %   | 70.19 %   | 0.6390 |
+| fastText-218 (NLLB)  | ~600 M |   218  | 59.53 %   | 71.64 %   | 0.6590 |
+| GlotLID v3           | ~600 M | 2 102  | 57.69 %   | 71.26 %   | 0.6729 |
+| **CommonLingua v7.2.1** | **2.35 M** | **334** | **77.63 %** | **82.92 %** | **0.7879** |
+
+CommonLingua reaches **+11.5 macro F1** over the next best baseline with **~250× fewer parameters**. The full per-language F1 breakdown ships in `eval_per_language.json`.
+
+### African subset
+
+CommonLID's African subset (17 languages with ≥ 100 gold samples — the regime where OpenLID/GlotLID/fastText reportedly underperform):
+
+| Model | African macro F1 |
+|---|---:|
+| OpenLID v2 | 0.5xx |
+| GlotLID v3 | 0.725 |
+| **CommonLingua v7.2.1** | **0.7222** |
+
+Notably, CommonLingua reaches **F1 = 0.975** on Amharic — a language Lingua does not support.
+
+### fp32 vs bf16
+
+The bf16 build is half the disk size and ~2.4× faster on H100, with **no measurable quality drop**: 0 of the 72 evaluated languages drift by more than 0.01 F1.
+
+| Build | Disk | Strict acc | Equiv acc | Macro F1 | African F1 | Lingua F1 |
+|---|---:|---:|---:|---:|---:|---:|
+| **fp32** (default) | 9.4 MB | 0.7763 | 0.8292 | **0.7879** | 0.7222 | 0.8806 |
+| **bf16**           | 4.7 MB | 0.7763 | 0.8292 | **0.7879** | 0.7221 | 0.8804 |
+| Δ                  | −50 %  | 0 | 0 | 0 | −0.0002 | −0.0003 |
+
+### Throughput
+
+Texts/sec (one paragraph = one text, ≤ 512 bytes input, padded). Real CommonLingua weights and the production code path:
+
+| Device | Setting | fp32 | bf16 | bf16 vs fp32 |
+|---|---|---:|---:|---:|
+| H100 80GB (bs=4096)             | best       | 10,962 | **26,236** | 2.4× |
+| H100 80GB (bs=1024)             |            | 10,892 | 26,130 | 2.4× |
+| H100 80GB (bs=256)              |            | 10,677 | 25,241 | 2.4× |
+| H100 80GB (bs=64)               | low-latency| 10,025 | 22,625 | 2.3× |
+| Sapphire Rapids CPU (8 threads) | bs=32      | _PENDING_ | _PENDING_ | _PENDING_ |
+| Sapphire Rapids CPU (1 thread)  | bs=32      | _PENDING_ | _PENDING_ | _PENDING_ |
+
+The press release that previously circulated cited "20 t/s on CPU, 3 000 t/s on GPU" — the actual GPU figure is **~9× higher** in fp32 and **~22× higher** in bf16. The bf16 build is recommended whenever the host supports it (essentially: anything Ampere or newer).
+
+## Training data
+
+Trained on **2,482,568 paragraphs across 334 languages**, drawn entirely from open-licensed and public-domain sources. Wikipedia provides the bulk (~93 %); the long tail is filled by Pralekha (Indic), VOA Africa, Cultural Heritage, OpenAlex (Indo-Malay journal data + African academic), Common Corpus adversarial pulls, Perseus / OpenPecha / eBible / Sefaria / Ben-Yehuda / Krike-Krake (ancient and minority-script corpora).
+
+Per-source contributions, license attribution, and full schema are documented in [PleIAs/CommonLingua-Train](https://huggingface.co/datasets/PleIAs/CommonLingua-Train).
+
+## Known limitations
+
+1. **Arabic dialect cluster** — Modern Standard (`arb`) is robust (F1 ≈ 0.95), but Moroccan (`ary`, F1 ≈ 0.47) and Egyptian (`arz`, F1 ≈ 0.25) Arabic are structurally hard: the dialects share large stretches of vocabulary with MSA and with each other. Not a data-volume problem; needs targeted corpus.
+2. **Indonesian / Malay (`ind` / `msa`)** — ~48 % msa error rate. Adding 20 k journal-provenance rows for each gave only marginal improvement; this pair will likely need supervised disambiguation features beyond byte-level signal.
+3. **Estonian (`est`) attractor** — accumulates ~750 false positives from unrelated languages. Model quirk to investigate; in practice a confidence threshold of 0.7 mostly removes spurious `est` predictions.
+4. **Lingala (`lin`)** — CommonLID's gold has the labels for `lin` reversed with Tiv/Yoruba (paper acknowledged a related labelling issue). Our F1 = 0 on this class is a benchmark artefact, not a model failure. Real-world `lin` predictions are correct on internal held-out data.
+5. **Short text (<50 chars)** — confidence drops sharply. The model is **not** intended for short-query LID; use CLD3 or a query-tuned model for that regime.
+
+## Files
+
+| File | Description |
+|---|---|
+| `model.pt`        | fp32 checkpoint (9.4 MB) |
+| `model.bf16.pt`   | bf16 checkpoint (4.7 MB) |
+| `model.py`        | ByteHybrid v2 architecture |
+| `predict.py`      | Standalone CLI (no `commonlingua` package required) |
+| `config.json`     | model config |
+| `lang2idx.json`   | 334-class label map |
+| `eval_per_language.json` | full per-language F1 on CommonLID |
+
+## Citation
+
+```bibtex
+@misc{commonlingua,
+  author = {{PleIAs}},
+  title  = {CommonLingua: Byte-level Language Identification for 334 Languages},
+  year   = {2026},
+  url    = {https://huggingface.co/PleIAs/CommonLingua}
+}
+```

config.json

@@ -0,0 +1,7 @@
+{
+  "config": "base_ngram",
+  "num_classes": 334,
+  "max_len": 512,
+  "epoch": 3,
+  "val_acc": 0.9615777881207785
+}

lang2idx.json

@@ -0,0 +1,336 @@
+{
+  "abk": 0,
+  "ace": 1,
+  "ach": 2,
+  "ady": 3,
+  "afr": 4,
+  "aka": 5,
+  "akk": 6,
+  "alt": 7,
+  "amh": 8,
+  "ami": 9,
+  "ang": 10,
+  "anp": 11,
+  "arb": 12,
+  "arc": 13,
+  "arg": 14,
+  "arq": 15,
+  "ary": 16,
+  "arz": 17,
+  "asm": 18,
+  "ast": 19,
+  "atj": 20,
+  "ava": 21,
+  "avk": 22,
+  "awa": 23,
+  "aym": 24,
+  "azb": 25,
+  "azj": 26,
+  "bak": 27,
+  "bam": 28,
+  "ban": 29,
+  "bar": 30,
+  "bbc": 31,
+  "bel": 32,
+  "ben": 33,
+  "bho": 34,
+  "bik": 35,
+  "bis": 36,
+  "bjn": 37,
+  "blk": 38,
+  "bod": 39,
+  "bos": 40,
+  "bpy": 41,
+  "bre": 42,
+  "bug": 43,
+  "bul": 44,
+  "bxr": 45,
+  "cat": 46,
+  "cbk": 47,
+  "cdo": 48,
+  "ceb": 49,
+  "ces": 50,
+  "che": 51,
+  "chr": 52,
+  "chu": 53,
+  "chv": 54,
+  "chy": 55,
+  "ckb": 56,
+  "cor": 57,
+  "cos": 58,
+  "crh": 59,
+  "csb": 60,
+  "cym": 61,
+  "dag": 62,
+  "dan": 63,
+  "deu": 64,
+  "din": 65,
+  "diq": 66,
+  "div": 67,
+  "dsb": 68,
+  "dty": 69,
+  "dzo": 70,
+  "egx": 71,
+  "egy": 72,
+  "ell": 73,
+  "eml": 74,
+  "eng": 75,
+  "epo": 76,
+  "est": 77,
+  "ett": 78,
+  "eus": 79,
+  "ewe": 80,
+  "ext": 81,
+  "fao": 82,
+  "fas": 83,
+  "fij": 84,
+  "fin": 85,
+  "fon": 86,
+  "fra": 87,
+  "fro": 88,
+  "frp": 89,
+  "frr": 90,
+  "fry": 91,
+  "ful": 92,
+  "fur": 93,
+  "gag": 94,
+  "gan": 95,
+  "gaz": 96,
+  "gcr": 97,
+  "gez": 98,
+  "gla": 99,
+  "gle": 100,
+  "glg": 101,
+  "glk": 102,
+  "glv": 103,
+  "gom": 104,
+  "gor": 105,
+  "got": 106,
+  "gpe": 107,
+  "grc": 108,
+  "gsw": 109,
+  "guc": 110,
+  "gug": 111,
+  "guj": 112,
+  "gur": 113,
+  "guw": 114,
+  "hak": 115,
+  "hat": 116,
+  "hau": 117,
+  "haw": 118,
+  "hbo": 119,
+  "hbs": 120,
+  "heb": 121,
+  "hif": 122,
+  "hin": 123,
+  "hrv": 124,
+  "hsb": 125,
+  "hun": 126,
+  "hye": 127,
+  "hyw": 128,
+  "ibo": 129,
+  "ido": 130,
+  "iku": 131,
+  "ile": 132,
+  "ilo": 133,
+  "ina": 134,
+  "ind": 135,
+  "inh": 136,
+  "ipk": 137,
+  "isl": 138,
+  "ita": 139,
+  "jam": 140,
+  "jav": 141,
+  "jbo": 142,
+  "jpn": 143,
+  "kaa": 144,
+  "kab": 145,
+  "kal": 146,
+  "kan": 147,
+  "kas": 148,
+  "kat": 149,
+  "kaz": 150,
+  "kbd": 151,
+  "kbp": 152,
+  "kcg": 153,
+  "khk": 154,
+  "khm": 155,
+  "kik": 156,
+  "kin": 157,
+  "kir": 158,
+  "kmr": 159,
+  "koi": 160,
+  "kom": 161,
+  "kon": 162,
+  "kor": 163,
+  "krc": 164,
+  "ksh": 165,
+  "lad": 166,
+  "lao": 167,
+  "lat": 168,
+  "latex": 169,
+  "lav": 170,
+  "lbe": 171,
+  "lez": 172,
+  "lfn": 173,
+  "lij": 174,
+  "lim": 175,
+  "lin": 176,
+  "lit": 177,
+  "lld": 178,
+  "lmo": 179,
+  "ltg": 180,
+  "ltz": 181,
+  "lug": 182,
+  "luo": 183,
+  "lzh": 184,
+  "mad": 185,
+  "mai": 186,
+  "mal": 187,
+  "mar": 188,
+  "mdf": 189,
+  "mhr": 190,
+  "min": 191,
+  "mkd": 192,
+  "mlg": 193,
+  "mlt": 194,
+  "mni": 195,
+  "mnw": 196,
+  "mri": 197,
+  "mrj": 198,
+  "msa": 199,
+  "mwl": 200,
+  "mya": 201,
+  "myv": 202,
+  "mzn": 203,
+  "nah": 204,
+  "nan": 205,
+  "nap": 206,
+  "nav": 207,
+  "nds": 208,
+  "nep": 209,
+  "new": 210,
+  "nia": 211,
+  "nld": 212,
+  "nno": 213,
+  "nor": 214,
+  "nov": 215,
+  "nqo": 216,
+  "nrf": 217,
+  "nso": 218,
+  "nya": 219,
+  "nyn": 220,
+  "oci": 221,
+  "olo": 222,
+  "orm": 223,
+  "ory": 224,
+  "oss": 225,
+  "pag": 226,
+  "pam": 227,
+  "pan": 228,
+  "pap": 229,
+  "pcd": 230,
+  "pcm": 231,
+  "pdc": 232,
+  "peo": 233,
+  "pfl": 234,
+  "pms": 235,
+  "pnb": 236,
+  "pnt": 237,
+  "pol": 238,
+  "por": 239,
+  "pus": 240,
+  "pwn": 241,
+  "quy": 242,
+  "rcf": 243,
+  "rmy": 244,
+  "roh": 245,
+  "ron": 246,
+  "rue": 247,
+  "run": 248,
+  "rup": 249,
+  "rus": 250,
+  "sah": 251,
+  "san": 252,
+  "sat": 253,
+  "scn": 254,
+  "sgs": 255,
+  "shi": 256,
+  "shn": 257,
+  "sin": 258,
+  "skr": 259,
+  "slk": 260,
+  "slv": 261,
+  "sme": 262,
+  "smn": 263,
+  "smo": 264,
+  "sna": 265,
+  "snd": 266,
+  "som": 267,
+  "sot": 268,
+  "spa": 269,
+  "sqi": 270,
+  "srd": 271,
+  "srn": 272,
+  "srp": 273,
+  "ssw": 274,
+  "stq": 275,
+  "sun": 276,
+  "sux": 277,
+  "swe": 278,
+  "swh": 279,
+  "szl": 280,
+  "szy": 281,
+  "tah": 282,
+  "tam": 283,
+  "tat": 284,
+  "tay": 285,
+  "tcy": 286,
+  "tel": 287,
+  "tet": 288,
+  "tgk": 289,
+  "tgl": 290,
+  "tha": 291,
+  "tir": 292,
+  "tly": 293,
+  "ton": 294,
+  "tpi": 295,
+  "trv": 296,
+  "tsn": 297,
+  "tso": 298,
+  "tuk": 299,
+  "tum": 300,
+  "tur": 301,
+  "txb": 302,
+  "tyv": 303,
+  "udm": 304,
+  "uig": 305,
+  "ukr": 306,
+  "urd": 307,
+  "uzb": 308,
+  "vec": 309,
+  "ven": 310,
+  "vep": 311,
+  "vie": 312,
+  "vls": 313,
+  "vol": 314,
+  "vro": 315,
+  "war": 316,
+  "wln": 317,
+  "wol": 318,
+  "wuu": 319,
+  "xal": 320,
+  "xcl": 321,
+  "xho": 322,
+  "xmf": 323,
+  "xog": 324,
+  "xto": 325,
+  "ydd": 326,
+  "yor": 327,
+  "yue": 328,
+  "zea": 329,
+  "zgh": 330,
+  "zha": 331,
+  "zho": 332,
+  "zul": 333
+}

model.bf16.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0a730271a5646cd7f547b4c631ddba181eaa434d113a7d08751625f39a0962bb
+size 4715426

model.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:63a2232378faefbde97c8269490793f0154109380488b4d6ae21f3839565f2f0
+size 9409133

model.py

@@ -0,0 +1,318 @@
+"""
+ByteHybrid v2: Byte-level document classifier with optional n-gram hash embeddings.
+
+Changes from v1:
+- Added ByteNgramEmbed: rolling hash of byte trigrams into fixed-size embedding table
+- New config "base_ngram" with ngram_buckets=4096, ngram_dim=64 (~262k extra params)
+- Backward compatible: existing configs work unchanged (ngram_buckets=0)
+"""
+
+import math
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+
+# ── Byte N-gram Hash Embedding ───────────────────────────────────────────
+
+
+class ByteNgramEmbed(nn.Module):
+    """Rolling hash of byte n-grams into fixed-size embedding table.
+    
+    Supports single n-gram size or multi-scale (e.g., trigrams + 5-grams).
+    Uses polynomial hash. Collisions act as regularization.
+    """
+    
+    def __init__(self, num_buckets=4096, embed_dim=64, n=3):
+        super().__init__()
+        self.n = n
+        self.num_buckets = num_buckets
+        self.embed = nn.Embedding(num_buckets, embed_dim)
+        
+    def _hash(self, byte_ids, n):
+        B, T = byte_ids.shape
+        clamped = byte_ids.clamp(max=255)
+        padded = F.pad(clamped, (0, n - 1), value=0)
+        h = torch.zeros(B, T, dtype=torch.long, device=byte_ids.device)
+        for i in range(n):
+            h = h * 257 + padded[:, i:i+T]
+        h = h % self.num_buckets
+        return h
+        
+    def forward(self, byte_ids):
+        return self.embed(self._hash(byte_ids, self.n))
+
+
+class MultiScaleNgramEmbed(nn.Module):
+    """Multi-scale n-gram hash embeddings (e.g., 3-gram + 5-gram).
+    
+    Each scale gets its own hash table and embedding. Outputs are summed.
+    """
+    
+    def __init__(self, num_buckets=4096, embed_dim=64, scales=(3, 5)):
+        super().__init__()
+        self.scales = scales
+        self.ngrams = nn.ModuleList([
+            ByteNgramEmbed(num_buckets, embed_dim, n=n) for n in scales
+        ])
+    
+    def forward(self, byte_ids):
+        out = self.ngrams[0](byte_ids)
+        for ng in self.ngrams[1:]:
+            out = out + ng(byte_ids)
+        return out
+
+
+# ── Causal Conv1d Block ──────────────────────────────────────────────────
+
+
+class ByteConvBlock(nn.Module):
+    """Causal conv1d + gated FFN. Captures local byte patterns."""
+
+    def __init__(self, d_model, kernel_size=15, expand=2):
+        super().__init__()
+        self.norm1 = nn.LayerNorm(d_model)
+        self.pad = kernel_size - 1
+        self.conv = nn.Conv1d(d_model, d_model, kernel_size, groups=d_model)
+        self.norm2 = nn.LayerNorm(d_model)
+        ffn_dim = d_model * expand
+        self.ffn_gate = nn.Linear(d_model, ffn_dim, bias=False)
+        self.ffn_up = nn.Linear(d_model, ffn_dim, bias=False)
+        self.ffn_down = nn.Linear(ffn_dim, d_model, bias=False)
+
+    def forward(self, x):
+        residual = x
+        x = self.norm1(x)
+        x = x.transpose(1, 2)
+        x = F.pad(x, (self.pad, 0))
+        x = F.silu(self.conv(x))
+        x = x.transpose(1, 2)
+        x = residual + x
+
+        residual = x
+        x = self.norm2(x)
+        x = self.ffn_down(F.silu(self.ffn_gate(x)) * self.ffn_up(x))
+        x = residual + x
+        return x
+
+
+# ── Attention Block ──────────────────────────────────────────────────────
+
+
+class ByteAttnBlock(nn.Module):
+    """Standard bidirectional attention + SwiGLU FFN with RoPE."""
+
+    def __init__(self, d_model, n_heads=4, expand=2):
+        super().__init__()
+        self.n_heads = n_heads
+        self.head_dim = d_model // n_heads
+
+        self.norm1 = nn.LayerNorm(d_model)
+        self.qkv = nn.Linear(d_model, 3 * d_model, bias=False)
+        self.out_proj = nn.Linear(d_model, d_model, bias=False)
+
+        self.norm2 = nn.LayerNorm(d_model)
+        ffn_dim = d_model * expand
+        self.ffn_gate = nn.Linear(d_model, ffn_dim, bias=False)
+        self.ffn_up = nn.Linear(d_model, ffn_dim, bias=False)
+        self.ffn_down = nn.Linear(ffn_dim, d_model, bias=False)
+
+    def forward(self, x):
+        B, T, D = x.shape
+        residual = x
+
+        x = self.norm1(x)
+        qkv = self.qkv(x).reshape(B, T, 3, self.n_heads, self.head_dim)
+        q, k, v = qkv.unbind(dim=2)
+        q = q.transpose(1, 2)
+        k = k.transpose(1, 2)
+        v = v.transpose(1, 2)
+
+        q, k = apply_rope(q, k, T, self.head_dim, x.device)
+
+        attn = (q @ k.transpose(-2, -1)) / (self.head_dim ** 0.5)
+        attn = attn.softmax(dim=-1)
+        out = (attn @ v).transpose(1, 2).contiguous().view(B, T, D)
+        out = self.out_proj(out)
+        x = residual + out
+
+        residual = x
+        x = self.norm2(x)
+        x = self.ffn_down(F.silu(self.ffn_gate(x)) * self.ffn_up(x))
+        x = residual + x
+        return x
+
+
+# ── Rotary Position Embedding ────────────────────────────────────────────
+
+
+def precompute_freqs(dim, max_len=4096, theta=10000.0):
+    freqs = 1.0 / (theta ** (torch.arange(0, dim, 2).float() / dim))
+    t = torch.arange(max_len)
+    freqs = torch.outer(t, freqs)
+    return torch.cos(freqs), torch.sin(freqs)
+
+
+def apply_rope(q, k, seq_len, head_dim, device):
+    cos, sin = precompute_freqs(head_dim, seq_len)
+    cos = cos[:seq_len].to(device=device, dtype=q.dtype)
+    sin = sin[:seq_len].to(device=device, dtype=q.dtype)
+
+    def rotate(x):
+        x1, x2 = x[..., : head_dim // 2], x[..., head_dim // 2 :]
+        return torch.cat([x1 * cos - x2 * sin, x2 * cos + x1 * sin], dim=-1)
+
+    return rotate(q), rotate(k)
+
+
+# ── Full Model ───────────────────────────────────────────────────────────
+
+
+class ByteHybrid(nn.Module):
+    """Byte-level classifier with optional n-gram hash embeddings.
+
+    Args:
+        num_classes: number of output classes
+        d_model: hidden dimension
+        n_conv: number of conv1d blocks
+        n_attn: number of attention blocks
+        n_heads: attention heads
+        max_len: maximum byte sequence length
+        conv_kernel: conv1d kernel size
+        ngram_buckets: hash table size for n-gram embeddings (0 = disabled)
+        ngram_dim: embedding dimension for n-gram hashes
+    """
+
+    def __init__(
+        self,
+        num_classes=13,
+        d_model=256,
+        n_conv=3,
+        n_attn=1,
+        n_heads=4,
+        ffn_expand=2,
+        max_len=2048,
+        conv_kernel=15,
+        ngram_buckets=0,
+        ngram_dim=64,
+        ngram_scales=None,
+    ):
+        super().__init__()
+        self.max_len = max_len
+
+        # Byte embedding: 256 possible byte values + 1 padding
+        self.embed = nn.Embedding(257, d_model, padding_idx=256)
+
+        # Optional n-gram hash embedding
+        # ngram_scales: tuple of n-gram sizes, e.g. (3,) or (3, 5)
+        self.ngram_embed = None
+        if ngram_buckets > 0:
+            scales = ngram_scales if ngram_scales else (3,)
+            if len(scales) == 1:
+                self.ngram_embed = ByteNgramEmbed(ngram_buckets, ngram_dim, n=scales[0])
+            else:
+                self.ngram_embed = MultiScaleNgramEmbed(ngram_buckets, ngram_dim, scales=scales)
+            self.ngram_proj = nn.Linear(ngram_dim, d_model, bias=False)
+
+        # Conv blocks
+        self.conv_layers = nn.ModuleList([
+            ByteConvBlock(d_model, kernel_size=conv_kernel, expand=ffn_expand)
+            for _ in range(n_conv)
+        ])
+
+        # Attention blocks
+        self.attn_layers = nn.ModuleList([
+            ByteAttnBlock(d_model, n_heads, ffn_expand)
+            for _ in range(n_attn)
+        ])
+
+        self.final_norm = nn.LayerNorm(d_model)
+
+        # Classification head
+        self.head = nn.Sequential(
+            nn.Linear(d_model, d_model),
+            nn.GELU(),
+            nn.Dropout(0.1),
+            nn.Linear(d_model, num_classes),
+        )
+
+    def forward(self, byte_ids):
+        """
+        Args:
+            byte_ids: (B, T) long tensor of byte values [0-255], padded with 256
+        Returns:
+            logits: (B, num_classes)
+        """
+        pad_mask = byte_ids != 256
+
+        x = self.embed(byte_ids)
+
+        # Add n-gram features if enabled
+        if self.ngram_embed is not None:
+            ng = self.ngram_embed(byte_ids)
+            x = x + self.ngram_proj(ng)
+
+        for layer in self.conv_layers:
+            x = layer(x)
+
+        for layer in self.attn_layers:
+            x = layer(x)
+
+        x = self.final_norm(x)
+
+        mask = pad_mask.unsqueeze(-1).to(x.dtype)
+        x = (x * mask).sum(dim=1) / mask.sum(dim=1).clamp(min=1)
+
+        return self.head(x)
+
+    @staticmethod
+    def encode_text(text, max_len=2048):
+        """Convert text string to byte tensor, padded to max_len."""
+        raw = text.encode("utf-8", errors="replace")[:max_len]
+        byte_ids = list(raw) + [256] * (max_len - len(byte_ids))
+        return torch.tensor(byte_ids, dtype=torch.long)
+
+
+# ── Configurations ───────────────────────────────────────────────────────
+
+CONFIGS = {
+    # ~2M params: 3 conv + 1 attn, d=256 (original)
+    "base": dict(d_model=256, n_conv=3, n_attn=1, n_heads=4, conv_kernel=15),
+    # ~2.3M params: base + trigram hash embeddings (4k buckets × 64 dim)
+    "base_ngram": dict(d_model=256, n_conv=3, n_attn=1, n_heads=4, conv_kernel=15,
+                       ngram_buckets=4096, ngram_dim=64),
+    # ~2.5M params: larger hash table
+    "base_ngram_large": dict(d_model=256, n_conv=3, n_attn=1, n_heads=4, conv_kernel=15,
+                             ngram_buckets=8192, ngram_dim=64),
+    # ~3.5M params: 3 conv + 2 attn, d=256
+    "large": dict(d_model=256, n_conv=3, n_attn=2, n_heads=4, conv_kernel=15),
+    # ~2M params: deeper conv, no attn
+    "conv_only": dict(d_model=256, n_conv=5, n_attn=0, n_heads=4, conv_kernel=15),
+    # ~2M params: wider kernel conv
+    "wide_conv": dict(d_model=256, n_conv=3, n_attn=1, n_heads=4, conv_kernel=31),
+    # Scaled-up configs
+    "d384": dict(d_model=384, n_conv=3, n_attn=1, n_heads=4, conv_kernel=15),
+    "d384_2attn": dict(d_model=384, n_conv=3, n_attn=2, n_heads=4, conv_kernel=15),
+    "d512": dict(d_model=512, n_conv=3, n_attn=1, n_heads=8, conv_kernel=15),
+    # 4-gram variant
+    "base_4gram": dict(d_model=256, n_conv=3, n_attn=1, n_heads=4, conv_kernel=15,
+                       ngram_buckets=4096, ngram_dim=64, ngram_scales=(4,)),
+    # Multi-scale: 3-gram + 5-gram (two hash tables, summed)
+    "base_multiscale": dict(d_model=256, n_conv=3, n_attn=1, n_heads=4, conv_kernel=15,
+                            ngram_buckets=4096, ngram_dim=64, ngram_scales=(3, 5)),
+    # Multi-scale: 3-gram + 4-gram + 5-gram
+    "base_multiscale3": dict(d_model=256, n_conv=3, n_attn=1, n_heads=4, conv_kernel=15,
+                             ngram_buckets=4096, ngram_dim=64, ngram_scales=(3, 4, 5)),
+}
+
+
+def count_params(model):
+    return sum(p.numel() for p in model.parameters())
+
+
+if __name__ == "__main__":
+    for name, cfg in CONFIGS.items():
+        model = ByteHybrid(num_classes=334, max_len=512, **cfg)
+        byte_ids = torch.randint(0, 256, (4, 512))
+        logits = model(byte_ids)
+        print(f"{name:<20s} {count_params(model):>10,} params  output={logits.shape}")

predict.py

@@ -0,0 +1,96 @@
+"""Standalone CommonLingua predict — single file, no `commonlingua` package required.
+
+Drop this next to `model.py` and the checkpoint, then:
+
+    python predict.py "Wikipedia is a free online encyclopedia, ..."
+    python predict.py --file input.txt
+    cat texts.tsv | python predict.py --stdin
+
+For the full Python API and parquet batch mode, install the package:
+
+    pip install "git+https://github.com/PleIAs/bytehybrid-lid#egg=commonlingua[hub]"
+"""
+import argparse
+import sys
+from pathlib import Path
+
+import numpy as np
+import torch
+
+sys.path.insert(0, str(Path(__file__).parent))
+from model import ByteHybrid, CONFIGS  # noqa: E402
+
+
+def load(checkpoint, dtype="fp32", device=None):
+    if device is None:
+        device = "cuda" if torch.cuda.is_available() else "cpu"
+    device = torch.device(device)
+    ckpt = torch.load(checkpoint, map_location="cpu", weights_only=False)
+    cfg = CONFIGS[ckpt["config"]]
+    model = ByteHybrid(num_classes=ckpt["num_classes"], max_len=ckpt["max_len"], **cfg)
+    model.load_state_dict(ckpt["model_state_dict"])
+    if dtype == "bf16":
+        model = model.to(torch.bfloat16)
+    model.eval().to(device)
+    idx2lang = {v: k for k, v in ckpt["lang2idx"].items()}
+    return model, idx2lang, ckpt["max_len"], device
+
+
+def encode(texts, max_len):
+    out = np.full((len(texts), max_len), 256, dtype=np.int64)
+    for i, t in enumerate(texts):
+        if not isinstance(t, str):
+            t = "" if t is None else str(t)
+        raw = t.encode("utf-8", errors="replace")[:max_len]
+        if raw:
+            out[i, :len(raw)] = np.frombuffer(raw, dtype=np.uint8)
+    return torch.from_numpy(out)
+
+
+@torch.no_grad()
+def predict(model, texts, idx2lang, max_len, device, top_k=3, batch_size=256):
+    out = []
+    for i in range(0, len(texts), batch_size):
+        chunk = texts[i:i + batch_size]
+        b = encode(chunk, max_len).to(device)
+        probs = torch.softmax(model(b).float(), dim=-1)
+        top_p, top_idx = probs.topk(top_k, dim=-1)
+        for p_row, idx_row in zip(top_p.cpu().tolist(), top_idx.cpu().tolist()):
+            out.append([(idx2lang[j], float(p)) for p, j in zip(p_row, idx_row)])
+    return out
+
+
+def main():
+    p = argparse.ArgumentParser()
+    p.add_argument("text", nargs="*", help="Texts to classify (one per arg).")
+    p.add_argument("--file", help="Read a single text from FILE.")
+    p.add_argument("--stdin", action="store_true", help="One text per line from stdin.")
+    p.add_argument("--checkpoint", default=str(Path(__file__).parent / "model.pt"))
+    p.add_argument("--dtype", choices=["fp32", "bf16"], default="fp32")
+    p.add_argument("--device", default=None)
+    p.add_argument("--top-k", type=int, default=3)
+    p.add_argument("--batch-size", type=int, default=256)
+    args = p.parse_args()
+
+    if args.stdin:
+        texts = [line.rstrip("\n") for line in sys.stdin if line.strip()]
+    elif args.file:
+        texts = [Path(args.file).read_text(encoding="utf-8")]
+    elif args.text:
+        texts = args.text
+    else:
+        p.print_help()
+        return
+
+    model, idx2lang, max_len, device = load(args.checkpoint, dtype=args.dtype, device=args.device)
+    print(f"# {len(idx2lang)} languages, max_len={max_len}, dtype={args.dtype}, device={device}",
+          file=sys.stderr)
+    results = predict(model, texts, idx2lang, max_len, device, args.top_k, args.batch_size)
+    for text, top in zip(texts, results):
+        preview = text[:80].replace("\n", " ")
+        others = " ".join(f"{lg}={p:.3f}" for lg, p in top[1:])
+        print(f"{top[0][0]}\t{top[0][1]:.4f}\t{others}\t{preview}")
+
+
+if __name__ == "__main__":
+    main()