File size: 3,736 Bytes
d72272a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 | # CanLex retrieval — precision investigation (2026-05-21)
Investigation of the persistent eval misses, with a tested, recommended fix.
**No retrieval-algorithm change has been deployed** — this is for review.
## The question
The eval had a handful of persistent misses where the correct provision ranked
outside the top 5. Why, and what fixes it?
## Diagnosis
Stage-by-stage trace of each miss — the gold provision's rank out of each
retriever, and after fusion:
| Query | Gold | BM25 rank | Semantic rank | Fused rank |
|---|---|---|---|---|
| pre-removal risk assessment | IRPA s.112 | 45 | 35 | 35 |
| report to a customs officer on arrival | Customs Act s.11 | 51 | **1** | 6 |
| duty to report imported goods | Customs Act s.12 | 58 | **1** | 6 |
| report large amounts of currency | PCMLTFA s.12 | 82 | 32 | 63 |
| seize unreported currency | PCMLTFA s.18 | 51 | **3** | 14 |
Two distinct causes:
**1. BM25 dilutes strong semantic hits.** For Customs Act s.11 and s.12 and
PCMLTFA s.18 the *semantic* retriever ranks the gold #1, #1, #3 — essentially
perfect. But BM25 ranks the same provision #51, #58, #51, because the query
keywords ("report", "currency", "arriving") are common words with no
distinctive term to latch onto. Reciprocal-rank fusion with equal weight
averages the two rankings, so a #1 semantic hit fused with a #51 BM25 hit lands
around #6. The strong signal is diluted by the weak one.
**2. The enacting statute is out-competed by elaborating material.** IRPA s.112
(PRRA) is ranked only mediocre by *both* retrievers (BM25 #45, semantic #35):
the IRPR regulations (s.160 "Application for protection", s.161, s.165, s.232)
elaborate the PRRA process across many focused sections, and the
currency-forfeiture case law (Dokaj, Williams, Hociung) crowds PCMLTFA s.12. One
enacting section cannot out-rank a dozen elaborating chunks on a topical query.
The `_ensure_legislation` guarantee added this batch mitigates this at the
production default `top_k=6` (PCMLTFA s.18 reaches #2 there, vs #11 at the
eval's `top_k=20`), but does not fix cause #2 fully.
## Tested fix — up-weight the semantic retriever
`canlex/index.py` now has a `W_SEM` constant: the weight on the semantic
retriever's contribution to the RRF fusion (default **1.0** = equal weight =
current, unchanged behaviour). Sweep on the 89-question eval set:
| W_SEM | Hit@1 | Hit@3 | Hit@5 | Hit@10 | MRR |
|---|---|---|---|---|---|
| 1.0 (current) | 0.573 | 0.787 | 0.876 | 0.921 | 0.701 |
| 1.5 | 0.629 | 0.798 | 0.888 | 0.933 | 0.737 |
| 2.0 | 0.652 | 0.809 | 0.899 | 0.933 | 0.752 |
| 3.0 | 0.652 | 0.820 | 0.910 | 0.933 | 0.754 |
Up-weighting the semantic retriever improves every metric monotonically, with no
regression — the gain is largest exactly where the diagnosis predicted
(Hit@1 +0.08, MRR +0.05).
## Recommendation
**Set `W_SEM = 2.0`** in `canlex/index.py`. It captures most of the gain
(Hit@1 0.57 -> 0.65, Hit@5 0.88 -> 0.90, MRR 0.70 -> 0.75) while keeping a
meaningful BM25 contribution. W_SEM=3.0 squeezes slightly more but tilts the
fusion heavily toward semantic; 2.0 is the balanced choice.
To apply: change the one constant, run `py -m canlex.eval` to confirm, redeploy.
Caveat: measured on the 89-question eval. Semantic up-weighting is principled
(the diagnostic shows semantic genuinely ranks these golds well), but keep an
eye on exact-keyword and section-number lookups after adopting it.
## Still hard after W_SEM=2.0
IRPA s.112 (PRRA) — cause #2 above; W_SEM does not fix it, because semantic
itself ranks s.112 only #35. A later option: an Act-over-its-own-regulation
tie-break, or accepting that the IRPR PRRA regulations are themselves a
reasonable answer and broadening that gold.
|