docs/QUESTION-AWARE-FRAMING.md

# Question-aware briefing framing

Diagnosis + recommendation for WS3 of the 2026-05-06 overnight pass.

The four-section briefing structure (Status / Empirical / Modeled /
Policy) is non-negotiable — it's what the four Mellea grounding checks
score, and rewriting it risks the 4/4 pass rate. What we want to change
is the **opening sentence of the Status section**, so it engages the
question shape the user actually asked. Today every briefing leads
with a generic "this address is exposed to flood risk" no matter
whether the user asked "should I worry?" (resident), "is disclosure
required?" (attorney), or "where should we prioritize hardening?"
(planner).

## Where the system_prompt is set today

| Call site | Path | `EXTRA_SYSTEM_PROMPT` source |
|-----------|------|------------------------------|
| `app/fsm.py:983 step_reconcile` | single_address (strict) | `app/reconcile.py:53` |
| `app/intents/neighborhood.py:377` | neighborhood (strict) | local @ `app/intents/neighborhood.py:35` |
| `app/intents/development_check.py:218` | development_check (strict) | local @ `app/intents/development_check.py:32` |
| `app/intents/live_now.py:212` | live_now (non-strict) | local @ `app/intents/live_now.py:38` |
| `app/reconcile.py:1089 reconcile()` | legacy non-strict | `app/reconcile.py:53` |

All four strict paths funnel into `mellea_validator.reconcile_strict_streaming(doc_msgs, system_prompt, ...)`. The system_prompt is currently a constant per call site.

## Three options Adam outlined

### (a) Planner sub-classifier
Add a fifth `question_type` field to the planner's JSON schema. Granite
4.1:3b classifies it alongside `intent`. Capstone reads it and conditions
the opening.

- ✅ Reuses an LLM that already understands the query
- ❌ Re-validates the planner contract — the `_validate()` parser, the
      schema doc, the fallback logic, and `scripts/probe_addresses.py`
      all need to grow a new field
- ❌ Costs another planner call iteration to converge if the model
      mis-emits the new field
- ❌ The planner is the warm-cache path the demo lives or dies on —
      changing its output schema five days before pitch is high-risk

### (b) Capstone prompt-conditional
Detect `question_type` from the raw query string with a deterministic
regex-based heuristic, augment the system_prompt with a per-type
"opening directive," pass through to `reconcile_strict_streaming`. No
planner change.

- ✅ Lowest blast radius — only touches the Capstone call sites
- ✅ Deterministic, testable, zero added latency (no LLM call)
- ✅ Easy to roll back — remove the `augment_system_prompt(...)` call
- ✅ The four Mellea grounding checks stay byte-identical
- ⚠️ Question-shape detection is heuristic, not learned. Edge cases
     (weird phrasings, code-switching) will fall back to a generic
     directive. Acceptable for the demo personas — they're known up
     front.

### (c) Both
Planner emits a hint, Capstone uses it as a tiebreaker over the
heuristic.

- Same risks as (a). Pre-demo, the marginal accuracy isn't worth the
  schema change.

## Recommendation: **option (b)**

Implementation lives in a new module `app/framing.py`:

- `detect(query, intent) -> question_type` — regex-based detector that
  returns one of 11 question types (the same eleven as the suite's
  framing rubric).
- `opening_instruction(question_type) -> str | None` — returns the
  directive sentence to inject, or None for `generic_exposure` (the
  default — current behavior unchanged).
- `augment_system_prompt(base, query, intent) -> str` — wraps the base
  prompt with a `QUESTION-AWARE OPENING` block.

Wiring:

1. `app/fsm.py` — add `set_query(q)` / `_current_query()` threadlocals
   alongside the existing `set_strict_mode`. `step_reconcile()` reads
   the query + intent to augment the system prompt before calling
   `reconcile_strict_streaming`.
2. `app/intents/single_address.py:run()` — call `set_query(query)`
   before `iter_steps`, reset in `finally` (matches the existing
   threadlocal pattern).
3. `app/intents/neighborhood.py:run()` — augment the local
   `EXTRA_SYSTEM_PROMPT` directly before passing to
   `reconcile_strict_streaming`.
4. `app/intents/development_check.py:run()` — same as neighborhood.
5. `app/intents/live_now.py:run()` — same; non-strict path so it just
   prepends to the system message content.
6. `app/reconcile.py:reconcile()` (legacy) — out of scope; it's not on
   the demo path and the strict path covers all current intents.

## Stop conditions

Per Adam's instruction: if the framing rubric scores below 3 on more
than five queries after the change lands, document what option (a) /
(c) would require and stop. **Do not silently expand scope.**

The "below 3 on more than five" test is the trigger to move to
heavier interventions — typically that the regex detector misclassified
the question or the Granite model is ignoring the directive under the
existing system prompt's strong four-section discipline.

---

## Outcome of the 2026-05-06 framed run

`tests/integration/results/2026-05-06/FRAMING-DELTA.md` is the full
report. Headline:

- Mean framing **2.25 → 2.80** (+0.55).
- Queries reaching 5/5: **0 → 3** — q01 resident habitability
  ("Yes, this address is exposed..."), q02 attorney disclosure
  ("Disclosure is warranted..."), q13 grant evidence
  ("Vulnerability assessment: ...").
- Queries reaching ≥ 4/5: **2 → 5**.
- Mellea grounding: 4 queries improved (3/4 → 4/4); 2 regressed
  (q01 4/4 → 3/4, q06 3/4 → 2/4); 14 unchanged. Net +2.

**Stop condition fired.** 12 / 20 framed queries scored below 3.
Triage of the 12:

1. **Rubric-vs-directive vocabulary mismatch (4 queries).** q03, q08,
   q10, q12 are bare neighborhood names that the suite labels
   `capital_planning`. The detector returns `journalism` (the
   bare-neighborhood fallback). Both are valid persona framings; the
   journalism directive *is* applied (the openings change), but the
   capital-planning rubric scores against verdict words like
   "prioritize" / "merits prioritization" that the journalism
   directive doesn't request. **Not a framing failure — a
   measurement asymmetry.**
2. **Short-prose floor (4 queries).** q07, q14, q15, q19 returned
   ≤ 200 chars of prose because the geocoder failed (q07, q14, q18 —
   long conversational queries) or the planner / NTA resolver
   short-circuited (q15 ranking query, q19 BBMCR project name).
   Documented in `OVERNIGHT-2026-05-06-OUT-OF-SCOPE.md`. No framing
   change can salvage these — they need geocoder + intent-router
   work first.
3. **Granite ignored the directive (4 queries).** q04 (bare address,
   underwriting label), q05 (bare borough, journalism label), q11
   (PS 188 ambiguous), q17 (compare intent), q20 (Astoria control).
   In each case the framing prompt was injected but the opening
   stayed generic. Granite 4.1's existing four-section discipline
   appears to overpower a soft "QUESTION-AWARE OPENING" directive
   for some question types; the verdict-style types (Yes/No,
   Disclosure, Vulnerability assessment) succeed because they have
   explicit token shapes the model can latch onto.

## What option (a) would require

Adam's instruction: if the stop condition fires, document option (a)
or (c) and stop — do not silently expand scope. **NOT IMPLEMENTED.**
Sketch:

1. **Planner schema gains a `question_type` field.** Add to
   `app/planner.py:PLAN_SCHEMA_DESC`, `Plan` dataclass, and
   `_validate()` so the model emits an 11-value enum alongside
   `intent`.
2. **Few-shot the planner on question_type.** Add 6-10 worked
   examples to `SYSTEM_PROMPT` (one per persona from RESEARCH.md)
   so granite4.1:3b reliably emits the right enum value. The
   planner is already running with `format=json` constrained
   decoding, so this is a pure prompt-engineering change.
3. **Capstone consumes the planner's question_type instead of the
   detector's.** `app.framing.augment_system_prompt` already takes
   `intent`; add a third `question_type` parameter that overrides
   `detect()` when present. Capstone callers (fsm.step_reconcile,
   the three intents) read it from `plan.question_type` and pass
   through.
4. **Fall back to the regex detector when the planner emits an
   unknown / missing value.** Belt-and-suspenders against planner
   regression.
5. **Re-validate** with the same 20-query suite. If mean framing
   moves from 2.80 → ≥ 3.5 (target: ≥ half the queries scoring 4+),
   option (a) was the right call. If not, the issue is downstream
   (Granite ignoring the directive); option (c) won't help.

**Cost estimate.** ~2-3 hr of work, plus re-validation against the
address probe + the 20-query suite. The risk is the planner
regressing on intent classification when prompted to also emit a
new field — Granite 4.1:3b at temperature 0 with constrained
decoding is robust but not infallible. Validate against the full
address probe before merging.

## What option (c) would add

Layer (a) on top of (b). When the planner emits a question_type that
matches the detector's, both agree → use the directive. When they
disagree → log the disagreement (telemetry), use the planner's.
Marginal value over (a) alone is small; defer unless (a) shows
misclassification on the 20-query suite.