Add BENCH-002 FLORES-200 results; update domain limitations caveat

README.md

@@ -45,11 +45,15 @@ This model is the first open neural machine translation system for Sango.
 | Hardware       | A100-SXM4-40GB (42.4 GB VRAM)               |
 | Training time  | 7.25 hours                                  |
 
-## Benchmark — BENCH-001
+## Benchmarks
 
-Evaluated on a held-out sample of the quality-filtered NLLB Sango-French corpus (N=200 sentences, LASER score ≥ 1.0, target language ID confidence ≥ 0.9). Evaluation script: `scripts/training/evaluate_model.py`.
+> **Note on BLEU for Sango**: BLEU penalises fluent paraphrases. chrF is more informative for Sango's morphological structure. Both are reported for completeness.
 
-### French → Sango
+### BENCH-001 — In-distribution (NLLB val.jsonl, N=200)
+
+Evaluated on a held-out sample of the quality-filtered NLLB Sango-French corpus (N=200 sentences, LASER score ≥ 1.0, target language ID confidence ≥ 0.9). **Same distribution as training data** — measures specialised quality on the task the model was trained for.
+
+#### French → Sango
 
 | Model                                       | BLEU      | chrF      |
 | ------------------------------------------- | --------- | --------- |
@@ -58,7 +62,7 @@ Evaluated on a held-out sample of the quality-filtered NLLB Sango-French corpus
 | Foundation LLM (cloud inference, reference) | 2.92      | 26.45     |
 | **Delta (ours vs baseline)**                | **+5.70** | **+3.15** |
 
-### Sango → French
+#### Sango → French
 
 | Model                                      | BLEU      | chrF      |
 | ------------------------------------------ | --------- | --------- |
@@ -66,9 +70,35 @@ Evaluated on a held-out sample of the quality-filtered NLLB Sango-French corpus
 | **MEYNG/nllb-sango-finetuned-600m (ours)** | **18.63** | **35.78** |
 | **Delta (ours vs baseline)**               | **+9.10** | **+4.90** |
 
-Full benchmark results: `scripts/training/results/bench001_threeway.json`
+Full results: `scripts/training/results/bench001_threeway.json`
+
+---
+
+### BENCH-002 — Out-of-domain (FLORES-200 devtest, N=200) — _Added 2026-05-22_
+
+Evaluated on FLORES-200 devtest (Wikipedia-style sentences, bri25yu Parquet mirror). **Out-of-distribution** relative to NLLB training corpus — measures generalisation to formal/encyclopaedic text.
+
+#### French → Sango
+
+| Model                                      | BLEU  | chrF  |
+| ------------------------------------------ | ----- | ----- |
+| Vanilla NLLB-200-distilled-600M (baseline) | 6.11  | 30.23 |
+| MEYNG/nllb-sango-finetuned-600m (ours)     | 3.39  | 22.96 |
+| **Delta (ours vs baseline)**               | −2.72 | −7.27 |
+
+#### Sango → French
+
+| Model                                      | BLEU  | chrF  |
+| ------------------------------------------ | ----- | ----- |
+| Vanilla NLLB-200-distilled-600M (baseline) | 6.62  | 26.57 |
+| MEYNG/nllb-sango-finetuned-600m (ours)     | 6.68  | 25.30 |
+| **Delta (ours vs baseline)**               | +0.06 | −1.27 |
+
+**Finding**: The model overfit to the NLLB corpus style. On Wikipedia-style formal text it regresses below the baseline, including hallucination on long complex sentences. **Use this model for text stylistically similar to its training domain (news / conversational parallel text).** For general-purpose Sango translation, the vanilla NLLB-200 baseline is safer on out-of-domain inputs.
+
+This finding drives the next priority: a diverse multi-domain Sango corpus for v2 training.
 
-> **Note on BLEU scores for Sango**: BLEU penalises fluent paraphrases. chrF is more informative for Sango's morphological structure. Both metrics are reported for completeness. FLORES-200 evaluation is pending (gated dataset access required).
+Full results: `scripts/training/results/bench002_flores200.json`
 
 ## Usage
 
@@ -131,11 +161,12 @@ Full training script: `scripts/training/train_on_azure.py`
 
 ## Known limitations
 
-- **Coverage**: The NLLB Sango-French corpus is the only source of training data. Translation quality reflects the distribution of topics in that corpus (weighted toward news / general text).
+- **Domain specialisation** _(primary limitation)_: The model was trained exclusively on NLLB corpus text (web-crawled news and conversational parallel sentences). BENCH-002 (FLORES-200 devtest, 2026-05-22) shows it regresses below the vanilla baseline on Wikipedia-style formal text (FR→SAG: −2.72 BLEU). It also exhibits hallucination and repetition on long, complex out-of-domain sentences. **Best suited for:** news, conversational, and general informal French-Sango translation. **Not recommended for:** formal, legal, medical, or encyclopaedic text.
+- **Coverage**: The NLLB Sango-French corpus is the only source of training data. Translation quality reflects the distribution of topics in that corpus.
 - **Sango orthography**: Sango has some inconsistency in diacritic use across documents. The model inherits this from the training data.
 - **Low-resource baseline**: Sango is a very low-resource language. Even after fine-tuning, expect lower absolute BLEU than high-resource pairs. chrF is a more reliable quality indicator for Sango.
 - **SAG→EN**: This model was not trained on Sango-English pairs. For Sango→English, use a two-step pipeline: `sag_Latn → fra_Latn → eng_Latn`.
-- **Evaluation dataset**: BENCH-001 uses a held-out split of the training corpus distribution. Independent evaluation on FLORES-200 is pending.
+- **Evaluation**: BENCH-001 uses a held-out split of the training corpus (in-distribution). BENCH-002 uses FLORES-200 devtest (out-of-domain). See the Benchmarks section for both.
 
 ## Related resources