CanLex retrieval — precision investigation (2026-05-21)
Investigation of the persistent eval misses, with a tested, recommended fix. No retrieval-algorithm change has been deployed — this is for review.
The question
The eval had a handful of persistent misses where the correct provision ranked outside the top 5. Why, and what fixes it?
Diagnosis
Stage-by-stage trace of each miss — the gold provision's rank out of each retriever, and after fusion:
| Query | Gold | BM25 rank | Semantic rank | Fused rank |
|---|---|---|---|---|
| pre-removal risk assessment | IRPA s.112 | 45 | 35 | 35 |
| report to a customs officer on arrival | Customs Act s.11 | 51 | 1 | 6 |
| duty to report imported goods | Customs Act s.12 | 58 | 1 | 6 |
| report large amounts of currency | PCMLTFA s.12 | 82 | 32 | 63 |
| seize unreported currency | PCMLTFA s.18 | 51 | 3 | 14 |
Two distinct causes:
1. BM25 dilutes strong semantic hits. For Customs Act s.11 and s.12 and PCMLTFA s.18 the semantic retriever ranks the gold #1, #1, #3 — essentially perfect. But BM25 ranks the same provision #51, #58, #51, because the query keywords ("report", "currency", "arriving") are common words with no distinctive term to latch onto. Reciprocal-rank fusion with equal weight averages the two rankings, so a #1 semantic hit fused with a #51 BM25 hit lands around #6. The strong signal is diluted by the weak one.
2. The enacting statute is out-competed by elaborating material. IRPA s.112
(PRRA) is ranked only mediocre by both retrievers (BM25 #45, semantic #35):
the IRPR regulations (s.160 "Application for protection", s.161, s.165, s.232)
elaborate the PRRA process across many focused sections, and the
currency-forfeiture case law (Dokaj, Williams, Hociung) crowds PCMLTFA s.12. One
enacting section cannot out-rank a dozen elaborating chunks on a topical query.
The _ensure_legislation guarantee added this batch mitigates this at the
production default top_k=6 (PCMLTFA s.18 reaches #2 there, vs #11 at the
eval's top_k=20), but does not fix cause #2 fully.
Tested fix — up-weight the semantic retriever
canlex/index.py now has a W_SEM constant: the weight on the semantic
retriever's contribution to the RRF fusion (default 1.0 = equal weight =
current, unchanged behaviour). Sweep on the 89-question eval set:
| W_SEM | Hit@1 | Hit@3 | Hit@5 | Hit@10 | MRR |
|---|---|---|---|---|---|
| 1.0 (current) | 0.573 | 0.787 | 0.876 | 0.921 | 0.701 |
| 1.5 | 0.629 | 0.798 | 0.888 | 0.933 | 0.737 |
| 2.0 | 0.652 | 0.809 | 0.899 | 0.933 | 0.752 |
| 3.0 | 0.652 | 0.820 | 0.910 | 0.933 | 0.754 |
Up-weighting the semantic retriever improves every metric monotonically, with no regression — the gain is largest exactly where the diagnosis predicted (Hit@1 +0.08, MRR +0.05).
Recommendation
Set W_SEM = 2.0 in canlex/index.py. It captures most of the gain
(Hit@1 0.57 -> 0.65, Hit@5 0.88 -> 0.90, MRR 0.70 -> 0.75) while keeping a
meaningful BM25 contribution. W_SEM=3.0 squeezes slightly more but tilts the
fusion heavily toward semantic; 2.0 is the balanced choice.
To apply: change the one constant, run py -m canlex.eval to confirm, redeploy.
Caveat: measured on the 89-question eval. Semantic up-weighting is principled (the diagnostic shows semantic genuinely ranks these golds well), but keep an eye on exact-keyword and section-number lookups after adopting it.
Still hard after W_SEM=2.0
IRPA s.112 (PRRA) — cause #2 above; W_SEM does not fix it, because semantic itself ranks s.112 only #35. A later option: an Act-over-its-own-regulation tie-break, or accepting that the IRPR PRRA regulations are themselves a reasonable answer and broadening that gold.