Document class-aware drug dialogue addition

README.md

@@ -31,7 +31,7 @@ against them too, but only v3 has been validated.
 | File | Corpus | Order | Size (gz) | Target register |
 |---|---|---|---|---|
 | `en-general.arpa.gz` | 3× GigaSpeech `s` + 1× People's Speech `clean` (~22 M effective words) | 4 | 67 MB | Conversational English with cased + punctuated output |
-| `en-medical.arpa.gz` | 5× MTSamples clinical dictation + 2× synthetic clinical dialogue (~61 M effective words) | 3 | **17 MB** | Medical English — clinical dictation + patient↔doctor dialogue register |
+| `en-medical.arpa.gz` | 5× MTSamples + 2× synthetic clinical dialogue + 2× class-aware drug dialogue (~77 M effective words) | 3 | **17 MB** | Medical English — clinical dictation + patient↔doctor dialogue + specialty drug names |
 
 More languages / domains coming as they're validated.
 
@@ -62,11 +62,33 @@ No `en-general` base, no forum text, no journal abstracts. The result
 matches earlier layered variants on fluency and disease recall while
 being 3–4× smaller.
 
-Remaining gap: specialty-drug vocabulary. MTSamples doesn't emphasise
-drug names, and synthetic dialogue uses mostly lay names. A future
-iteration will inject template-generated drug-dialogue text
-(e.g. "I was prescribed {drug} for my {condition}" with RxNorm +
-ICD-10 fills) to close this gap without corrupting the speech register.
+## Specialty drug coverage via class-aware template dialogue
+
+MTSamples + the synthetic-dialogue corpus use mostly lay drug names
+("paracetamol", "ibuprofen"); specialty drugs (`sertraline`,
+`olanzapine`, `apixaban`, `paclitaxel`) appear rarely. Earlier log-prob
+probes showed weak per-token priors on these (−2.0 to −3.0, vs
+−0.7 to −1.0 for well-covered phrases).
+
+The current build adds 2× of template-generated speech-register drug
+dialogue, filled from a curated catalog of ~330 generic drug names
+across 35 WHO-ATC-style classes paired with class-appropriate
+conditions:
+
+    "started on lithium for bipolar disorder"
+    "I've been taking apixaban for my atrial fibrillation"
+    "patient is on paclitaxel for breast cancer"
+    "can I get a refill of my sertraline"
+
+Class-aware pairing means we don't produce implausible combinations
+like "lithium for sinusitis". This lifts specialty-drug log-probs to
+the -0.7 to -1.2 range (comparable to well-covered general phrases)
+without degrading general fluency.
+
+PriMock-57 validation ties the earlier variant on entity F1 (since
+PriMock audio doesn't contain specialty drug mentions to exercise the
+improved priors). Users with dictated psychiatry / oncology /
+cardiology content should see the benefit directly.
 
 ## Usage
 
@@ -101,11 +123,13 @@ score = lm.score("42 17 5", bos=False, eos=False)
 ```
 # Extract MTSamples (galileo-ai/medical_transcription_40, text column).
 # Extract CodCodingCode/cleaned-clinical-conversations, stripping
-#   DOCTOR:/PATIENT: prefixes and deduping turns across overlapping rows
-#   (scripts/kenlm_build/extract_synthetic_dialogue.py in the Vernacula repo).
+#   DOCTOR:/PATIENT: prefixes and deduping turns across overlapping rows.
+# Generate ~8M words of class-aware drug dialogue using the curated
+#   drug_classes.json catalog (see scripts/kenlm_build/ in Vernacula).
 # Layer:
 #   (for _ in 1..5; cat mtsamples.txt; end
-#    for _ in 1..2; cat synthetic-dialogue.txt; end) > en-medical.corpus.txt
+#    for _ in 1..2; cat synthetic-dialogue.txt; end
+#    for _ in 1..2; cat drug-dialogue.txt; end) > en-medical.corpus.txt
 # Tokenise with nvidia/parakeet-tdt-0.6b-v3 tokenizer.json, then
 #   lmplz --order 3 --prune "0 0 1" --discount_fallback
 # gzip the ARPA.