experiments/03_granite_reranker/RESULTS.md

Phase 3 — Granite Embedding Reranker R2 (cross-encoder, 149 M)

Status

Working end-to-end on the existing 5-PDF corpus, both backends.

Model

• Model: ibm-granite/granite-embedding-reranker-english-r2
• Type: cross-encoder reranker (149 M params)
• License: Apache-2.0 (verified, HF cardData)
• Loader: sentence_transformers.CrossEncoder — sidecar pattern, no vLLM --task score per project decision
• Library declared: sentence-transformers

Pipeline

1. rerank.py — loads the cross-encoder, scores [query, candidate] pairs, returns ranked top-K.
2. run_double_gate.py — calls the existing app.rag.retrieve_top_k-equivalent (with the per-doc dedup bypassed), gathers top-20, reranks to top-3, and runs both backends' reconciler against the top-1 passage.

Validation

Hand-crafted query + 5 candidate paragraphs

Query: "What are flood risks in Hollis, Queens?"

The reranker correctly ranked the Hollis-Ida paragraph #1 (score 0.93), Sandy/Brighton #2 (0.77), and Rockaways #3 (0.73). The Newtown Creek WWTP and Bluebelt operations paragraphs (off-topic for a Hollis flood-risk query) were correctly demoted out of the top-3.

Real corpus end-to-end

Query: "What flood risk does Hollis, Queens face from heavy rainfall?"

Retriever (Granite Embedding 278 M) top-3:

Rank	Retriever score	Doc	Excerpt
1	0.760	rag_mta	"Urgent Call for Action 7 Climate Resilience Roadmap…"
2	0.749	rag_comptroller	"Forecast & Emergency Plan Activation Flash flooding…"
3	0.749	rag_comptroller	"Is New York City Ready for Rain? An Investigation…"

Reranker top-3 (from retriever's top-20):

Rank	Reranker score	Was retriever rank	Doc	Excerpt
1	0.886	6	rag_comptroller	"Is New York City Ready for Rain?… (preparedness section)"
2	0.869	4	rag_comptroller	"Heavy rains persisted for more than an hour in southern Brooklyn…"
3	0.869	1	rag_mta	"Urgent Call for Action 7 Climate Resilience Roadmap…"

The reranker is doing its job: it surfaced a query-specific preparedness paragraph (originally rank 6 — buried by the retriever) and demoted a generic MTA boilerplate paragraph (originally rank 1) to position 3.

Honesty under uncertainty

Neither selected paragraph specifically mentions Hollis. Both backends correctly refused to invent a Hollis-specific answer and said so plainly with a citation:

Backend	Latency	Output
Ollama (M-series MPS)	10.56 s	"The provided document…does not specifically mention Hollis, Queens…I cannot determine the flood risk for Hollis, Queens from heavy rainfall…[rag_comptroller]"
vLLM (AMD MI300X)	0.68 s	"The provided document does not contain specific information about the flood risk faced by Hollis, Queens from heavy rainfall. [rag_comptroller]"

This is the desired silence-over-confabulation behavior. The reranker

• reconciler combination did not surface a false claim despite there being a temptation (the document discusses a 2024 storm in NYC generally).

Latency budget

Stage	Latency	Notes
Retriever (Granite Embedding 278 M) cold load + index	52.7 s	One-time at app boot; amortized in production
Retriever per-query	< 0.1 s	Already in production
Reranker cold load (149 M)	1.8 s	One-time at app boot
Reranker score 20 candidates	0.93 s	M3 Pro CPU, batched
Reconcile (Ollama, M-series)	10.6 s
Reconcile (vLLM, AMD MI300X)	0.7 s	~15× faster

The reranker adds ~1 s to the user-visible path on CPU. Negligible relative to the existing reconciler latency, well under the brief's demo budget.

Findings worth remembering

1. The retriever's per-doc dedup is in the wrong place. Currently app/rag.py:retrieve() keeps "at most 1 chunk per doc" and then returns top-K. For the reranker integration, this should be inverted: gather top-20 with duplicates, rerank, then dedup to top-3. Otherwise we're throwing away high-relevance chunks before the rerank ever sees them.

2. Cross-encoder cache_dir arg is deprecated in current sentence-transformers; passes through with a warning. Move to model_kwargs={"cache_dir": ...} when integrating to silence it.

3. Reranker disagrees with the retriever in interesting ways. On the test query the retriever's rank-1 (a generic MTA roadmap intro) was a content-light string that scored high on lexical/embedding surface similarity to "flood risk heavy rainfall". The reranker correctly surfaced more specific content. This is the canonical reason cross-encoder reranking matters.

4. Sidecar deployment story. No GPU needed for the reranker; ~600 MB resident on CPU; loads in ~2 s after first download. Fits trivially in the HF Spaces T4 image. The vLLM-served alternative was explicitly out-of-scope per the project decision and isn't needed for these latencies.

Files

03_granite_reranker/
  rerank.py           CrossEncoder load + predict wrapper
  run_double_gate.py  retriever -> reranker -> reconciler probe
  RESULTS.md          (this file)
  .cache/             reranker weights, double_gate_*.json

Conclusion

Specialist works on both backends with the expected behavior change (reranker reorders top-3 in a query-relevant way; reconciler refuses to fabricate when source content doesn't address the query).

Recommended path forward: integrate as a one-line addition to app/rag.py:retrieve(): take retriever top-K=20 (drop the existing per-doc dedup), call the reranker, then dedup to top-3. Load the cross-encoder once at app boot in warm(). Single env var RIPRAP_RERANKER_ENABLE=1 to gate the new behavior so the existing production path is unchanged by default.