experiments/02_gliner_extraction/RESULTS.md

Phase 2 — GLiNER (gliner_medium-v2.1) structured extraction

Status

Working end-to-end on a real corpus PDF, both backends.

Model

• Model: urchade/gliner_medium-v2.1 (151 M params)
• License: Apache-2.0 (verified — model card frontmatter; NOT the gliner_base variant which is CC-BY-NC-4.0).
• Loader: gliner.GLiNER.from_pretrained(...) — pure HF, no third-party fine-tune framework.

Pipeline

1. extract.py — loads GLiNER, runs predict_entities() on a paragraph with the 5 typed labels: nyc_location, dollar_amount, date_range, agency, infrastructure_project. Threshold 0.45 (tuned by inspection).
2. extract_from_pdf.py — pulls paragraph text from a corpus PDF via pypdf, runs GLiNER on the longest paragraphs.
3. emit_doc.py — packages the typed list into a role: "document gliner_<source>" chat message. doc_id format: gliner_comptroller, gliner_dep, etc.
4. run_double_gate.py — end-to-end on a corpus PDF + paired Ollama/vLLM probe.

Validation

Hand-crafted paragraph (sanity)

"The NYC Department of Environmental Protection allocated $5.6 million for the Bluebelt expansion in Hollis, Queens for fiscal year 2025-2027. The Newtown Creek wastewater treatment plant in Brooklyn will receive an additional $12 million from NYCHA's resilience fund."

GLiNER extracted 9/9 expected entities at score ≥ 0.59: [agency] NYC Department of Environmental Protection, [dollar_amount] $5.6 million, [infrastructure_project] Bluebelt expansion, [nyc_location] Hollis, Queens, [date_range] fiscal year 2025-2027, [infrastructure_project] Newtown Creek wastewater treatment plant, [nyc_location] Brooklyn, [dollar_amount] $12 million, [agency] NYCHA.

Real corpus PDF — comptroller_rain_2024.pdf

Running on the longest paragraph (~3 KB of text, methodology section):

• 15 entities extracted
• 13× agency (mostly DEP repeated, New York City Comptroller, Comptroller's Office)
• 1× date_range
• 1× nyc_location
• Two dollar_amount hits ($15,000, $22.5 million, $ 875 million) on a different paragraph in the same PDF (top --top 2)

Citation discipline is preserved: cited [gliner_comptroller] resolves to a real input doc_id, agency tags align to actual surface text, no hallucinated dollar amounts in either backend's output.

Double-gating

run_double_gate.py --pdf comptroller_rain_2024.pdf --source-id comptroller:

Backend	Latency	Cited content
Ollama (M-series MPS)	11.94 s	"The NYC Department of Environmental Protection (DEP) has committed to implementing flood mitigation measures as part of the city's preparedness for flash flooding, as detailed in the report by the Office of the NYC Comptroller Brad Lander [gliner_comptroller]."
vLLM (AMD MI300X)	0.58 s	"The NYC Department of Environmental Protection (DEP) has committed to implementing flood mitigation measures as part of the City's preparedness outlined in New Normal, Rainfall Ready, and Ida, as documented by the NYC Comptroller's Office in the source [gliner_comptroller]."

Both citations resolve correctly. vLLM is again ~20× faster than Ollama on this prompt size.

Findings worth remembering

1. Threshold tuning matters. Default GLiNER threshold=0.5 misses agency = "Comptroller's Office" (it scored 0.45). 0.45 catches it without producing many false positives in the policy corpus. Worth re-tuning per source PDF if integrated.

2. GLiNER is fast even on CPU. Per-paragraph extract is 0.3 s on M3 Pro. The model load itself is the dominant cost (6 s); in production it stays loaded, so per-call latency is sub-second.

3. No comparative reasoning over the extractions. GLiNER returns typed spans, not relations. The reconciler infers the relation ("DEP allocated $X for Y in Z") from co-occurrence in the paragraph. That's fine for our briefings since they are paragraph- scoped, but stronger relational extraction (REBEL, etc.) would need a different model.

4. The current ranker is a placeholder. extract_from_pdf.py ranks paragraphs by length, not query relevance. In production this specialist consumes the existing Granite Embedding 278M retriever's top-K rather than picking longest paragraphs.

Files

02_gliner_extraction/
  extract.py             GLiNER load + predict_entities wrapper
  extract_from_pdf.py    pypdf paragraph splitter + GLiNER pass
  emit_doc.py            build gliner_<source> doc message
  run_double_gate.py     end-to-end + Ollama/vLLM probe
  RESULTS.md             (this file)
  .cache/                GLiNER weights, double_gate_*.json

Conclusion

Specialist works on both backends. Recommended path forward: integrate as a wrapper over the existing app/rag.py retriever output — GLiNER runs on the top-3 retrieved paragraphs and emits one gliner_<source_pdf> doc per paragraph, with the source_id derived from the PDF filename slug. The wrapper does not replace rag.py; it adds typed structure to its output for the reconciler.