--- license: cc-by-4.0 language: - en tags: - kenlm - n-gram - language-model - speech-recognition - parakeet - tdt - asr - shallow-fusion - vernacula library_name: kenlm --- # KenLM n-gram LMs for Parakeet TDT shallow fusion Subword-level KenLM ARPAs built over the [`nvidia/parakeet-tdt-0.6b-v3`](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3) tokenizer, intended for use with [Vernacula](https://github.com/christopherthompson81/vernacula)'s Parakeet beam decoder via shallow LM fusion. Each file's "words" are Parakeet subword IDs (integers), not natural-language words — this lets the decoder score the LM at every beam expansion without any subword-to-word round-trip. Other Parakeet checkpoints share vocabulary layouts so these LMs may work against them too, but only v3 has been validated. ## Files | File | Corpus | Order | Size (gz) | Target register | |---|---|---|---|---| | `en-general.arpa.gz` | 3× GigaSpeech `s` + 1× People's Speech `clean` (~22 M effective words) | 4 | 67 MB | Conversational English with cased + punctuated output | | `en-medical.arpa.gz` | 5× MTSamples + 2× synthetic clinical dialogue + 2× class-aware drug dialogue (~77 M effective words) | 3 | **17 MB** | Medical English — clinical dictation + patient↔doctor dialogue + specialty drug names | More languages / domains coming as they're validated. ## Design: speech-register per-domain corpus Early iterations of `en-medical` mixed MTSamples (spoken dictation) with HealthCareMagic (patient-written forum Q&A) and PubMed abstracts (formal written prose), then layered that over an `en-general` base. This performed worse than plain `en-general` on medical-entity F1. Investigation showed shallow fusion is a **style predictor, not a knowledge predictor**: the LM biases the decoder toward sequences it's seen, and written-prose medical text pushes the decoder into patterns that don't match conversational clinical audio. The "general base" layer was helping purely because GigaSpeech and People's Speech are spoken transcripts — *not* because of their domain coverage. The current `en-medical` is therefore built entirely from spoken-register medical content: - **MTSamples** (~2.2 M words, natural clinical dictation: H&P, SOAP notes, op reports) — upweighted 5× to compensate for its small raw size. - **CodCodingCode/cleaned-clinical-conversations** (~25 M deduped words of synthetic doctor↔patient dialogue spanning dozens of specialties and presentations) — upweighted 2×. No `en-general` base, no forum text, no journal abstracts. The result matches earlier layered variants on fluency and disease recall while being 3–4× smaller. ## Specialty drug coverage via class-aware template dialogue MTSamples + the synthetic-dialogue corpus use mostly lay drug names ("paracetamol", "ibuprofen"); specialty drugs (`sertraline`, `olanzapine`, `apixaban`, `paclitaxel`) appear rarely. Earlier log-prob probes showed weak per-token priors on these (−2.0 to −3.0, vs −0.7 to −1.0 for well-covered phrases). The current build adds 2× of template-generated speech-register drug dialogue, filled from a curated catalog of ~330 generic drug names across 35 WHO-ATC-style classes paired with class-appropriate conditions: "started on lithium for bipolar disorder" "I've been taking apixaban for my atrial fibrillation" "patient is on paclitaxel for breast cancer" "can I get a refill of my sertraline" Class-aware pairing means we don't produce implausible combinations like "lithium for sinusitis". This lifts specialty-drug log-probs to the -0.7 to -1.2 range (comparable to well-covered general phrases) without degrading general fluency. PriMock-57 validation ties the earlier variant on entity F1 (since PriMock audio doesn't contain specialty drug mentions to exercise the improved priors). Users with dictated psychiatry / oncology / cardiology content should see the benefit directly. ## Usage ### In Vernacula (recommended) Settings → Speech Recognition → Language model → pick "English (General)". The app downloads the file automatically and wires it into the beam decoder. ### From the Vernacula CLI ```bash vernacula --audio sample.wav --model \ --lm en-general.arpa.gz \ --lm-weight 0.15 ``` Passing `--lm` auto-bumps beam width to 4 (fusion has no effect in greedy mode). Typical fusion weight is 0.1–0.3; 0.15 is the default used by Vernacula's Settings picker. ### Directly with the KenLM Python bindings ```python import kenlm lm = kenlm.Model("en-general.arpa.gz") # tokens must be Parakeet subword IDs stringified: score = lm.score("42 17 5", bos=False, eos=False) ``` ## How `en-medical.arpa.gz` was built ``` # Extract MTSamples (galileo-ai/medical_transcription_40, text column). # Extract CodCodingCode/cleaned-clinical-conversations, stripping # DOCTOR:/PATIENT: prefixes and deduping turns across overlapping rows. # Generate ~8M words of class-aware drug dialogue using the curated # drug_classes.json catalog (see scripts/kenlm_build/ in Vernacula). # Layer: # (for _ in 1..5; cat mtsamples.txt; end # for _ in 1..2; cat synthetic-dialogue.txt; end # for _ in 1..2; cat drug-dialogue.txt; end) > en-medical.corpus.txt # Tokenise with nvidia/parakeet-tdt-0.6b-v3 tokenizer.json, then # lmplz --order 3 --prune "0 0 1" --discount_fallback # gzip the ARPA. ``` Upstream corpora have public-redistribution precedent in the medical NLP literature; attribution is required under the CC-BY-4.0 umbrella of this derivative. ## How `en-general.arpa.gz` was built ``` # Corpus: # 3x GigaSpeech `s` subset (Apache 2.0, cased + punctuation-tagged) # 1x People's Speech `clean` subset (CC-BY-4.0, lowercase, no punct) # Rationale: People's Speech carries conversational-register priors # (backchannels, disfluencies). GigaSpeech carries the case + punctuation # style Parakeet's output expects. 3x upweight on GigaSpeech balances the # raw size asymmetry so the case signal survives without being drowned out. # Tokenised with nvidia/parakeet-tdt-0.6b-v3 tokenizer.json: # ~27M subwords across ~1M sentences. # Built with KenLM's lmplz: lmplz -o 4 --prune 0 0 1 1 --discount_fallback \ --vocab_estimate 8193 \ --text en-mixed.tok --arpa en-general.arpa gzip en-general.arpa ``` See Vernacula's [`scripts/kenlm_build/`](https://github.com/christopherthompson81/vernacula/tree/main/scripts/kenlm_build) for the exact scripts. ## Validation On a 600 s en-US conversational sample (157 VAD segments, held-out from the corpus), Vernacula's Parakeet decoder exhibits a known beam=4 multilingual-drift regression where an English backchannel "Uh uh. Ya." transcribes as Spanish "ajá, ya". With this LM fused at weight 0.15 the line recovers to "Uh uh, yeah." and ~500 other lines stay unchanged vs greedy decoding — proper nouns preserved, punctuation preserved. ## License & attribution This LM is a derivative of its training corpora. It's released under **CC-BY-4.0** (the union of its sources' terms). Upstream corpora (attribution required): - [MLCommons/peoples_speech](https://huggingface.co/datasets/MLCommons/peoples_speech) — transcripts portion, CC-BY-4.0. (en-general) - [speechcolab/gigaspeech](https://huggingface.co/datasets/speechcolab/gigaspeech) — transcripts portion, Apache 2.0. (en-general) - [galileo-ai/medical_transcription_40](https://huggingface.co/datasets/galileo-ai/medical_transcription_40) — MTSamples mirror, clinical dictation. (en-medical) - [CodCodingCode/cleaned-clinical-conversations](https://huggingface.co/datasets/CodCodingCode/cleaned-clinical-conversations) — synthetic (LLM-generated) doctor↔patient dialogues across a broad range of conditions and specialties. (en-medical) ## Caveats - Subword-ID-keyed. Not a word-level LM — you can't load it in a word-level ASR decoder without first mapping through Parakeet's tokenizer. - 4-gram with `--prune 0 0 1 1` (drops 3-grams and 4-grams seen exactly once). If you need tighter priors, rebuild with `--prune 0 0 0 0` at the cost of roughly 3× file size. - Best at capturing local lexical choice (e.g. "uh huh" vs "ajá"). Doesn't carry sentence-level priors that might rescue wholly-ambiguous short utterances (e.g. a standalone "ajá" with no context).