imultimodal sensing → real stylistic constraints

Affect, gestures, and air-writing now actually steer the LLM's
word choice instead of just being metadata. Each emotion maps to
a StyleDirective (register, prefer/avoid words, opener hint,
exemplar) that's rendered as explicit instructions in the
per-turn user message; gestures override the opener when present;
air-writing recognises 8 single-stroke shapes (yes, ?, hi, help,
done, more, water, stop) and both biases retrieval via bucket
keywords and gets incorporated verbatim by the planner.

Fixes along the way:
- LCP was measuring mouth x-drift, not vertical lip-corner pull.
Rewrote it as (mouth_centre.y - corner_avg.y) / inter_ocular
and retuned thresholds; FRUSTRATED now has a second trigger
path (brows lowered + squinting).
- Calibration now averages the first 30 frames instead of a
single-frame snapshot. Affect stays null during calibration
but gaze/gesture/air-writing still flow.
- Deepcopy the shared _AFFECT_CONFIG in both intent paths so
downstream mutations can't corrupt the module constant.
- compute_multimodal_alignment now returns non-zero scores
(affect via sentiment lexicon, gesture via opener regex,
gaze via retrieved-chunk bucket match).
- LLM temperature 0.4 → 0.8 so the sensing→output link is
actually visible in the response.

README.md

@@ -288,7 +288,7 @@ multimodal_aac_chatbot/
 │   │   ├── graph.py               run_pipeline() — plain function chain
 │   │   ├── state.py               PipelineState TypedDict
 │   │   └── nodes/                 intent, retrieval, planner, feedback
-│   ├── sensing/labels.py          GESTURE_TO_TAG (sensing runs in browser)
+│   ├── sensing/labels.py          GESTURE_DIRECTIVES (sensing runs in browser)
 │   ├── retrieval/                 BGE embeddings (torch tensor) + bucket priors
 │   ├── generation/llm_client.py   2-tier Ollama Cloud LLM client (primary/fallback)
 │   └── guardrails/checks.py      Input + output safety checks
@@ -340,7 +340,7 @@ Adding a new persona: drop a JSON file into `data/memories/` following the schem
 
 From the spec (pages 10–11). Tags: **[Core]** = must do, **[Bonus]** = nice to have, **[Eval]** = for the grade.
 
-Heads up: all camera/sensing stuff is in the frontend (MediaPipe JS). Backend just gets the labels (`affect`, `gesture_tag`, `gaze_bucket`). The `backend/sensing/` python modules are dead code.
+Heads up: all camera/sensing stuff is in the frontend (MediaPipe JS). Backend just gets the labels (`affect`, `gesture_tag`, `gaze_bucket`). Only `backend/sensing/labels.py` (`GESTURE_DIRECTIVES`) lives on the backend.
 
 ### Dataset
 
@@ -359,10 +359,13 @@ Heads up: all camera/sensing stuff is in the frontend (MediaPipe JS). Backend ju
   - [x] intent-aware turnaround: PERSONAL re-retrieves excluding the rejected bucket *and* exact rejected chunk texts (with `turnaround_min_score` floor — falls back to original chunks rather than degrading); PRESENT_STATE flips emotional read or admits uncertainty
   - [x] UI: rejected bubble gets strikethrough + "rephrased" badge, new bubble appended with "↻ turnaround" badge — both visible (you can't unsay something to a partner). Manual "↻ Not quite right" button as fallback
   - [x] guards: `turnaroundConsumedTurnRef` prevents self-retrigger loops; backend `turn_id` returned in `ChatResponse` so frontend doesn't desync on persona switch; stale-turn 409
-- [ ] **[Core]** Smile / positive affect should actually change the wording (more positive lexicon), not just be metadata. Right now it's annotated in the prompt but we never checked if the LLM is doing anything with it — probably need a stronger constraint or example in the prompt
-- [ ] **[Core]** Air-writing is treated as raw text appended to the query. Spec wants it as a stylistic constraint too — should it bias tone, or stay query-only? Decide and document
+- [x] **[Core]** Smile / positive affect actually changes wording now. Affect compiles into a `StyleDirective` (register + prefer/avoid words + exemplar + opener hint) rendered as explicit instructions in the turn-specific user message — see `_AFFECT_CONFIG` in [backend/pipeline/nodes/intent.py](backend/pipeline/nodes/intent.py) and `_build_user` in [backend/pipeline/nodes/planner.py](backend/pipeline/nodes/planner.py). The persona's own `stylistic_preferences` (from the memory JSONs) carry the stable baseline in the cached system message; the affect directive is how that baseline shifts per turn. Measured by `compute_multimodal_alignment` (positive/negative lexicon).
+  - Fixed a long-standing bug where LCP (lip-corner pull) was accidentally the *x-coordinate* of the mouth centre, so it drifted on head turns and almost never fired FRUSTRATED. Now measured as vertical pull of the corners relative to mouth centre, normalised by inter-ocular distance. HAPPY/FRUSTRATED thresholds retuned to the new scale; FRUSTRATED also triggers on brows-lowered + squinting as a second path. See `computeAffectVector` and `classifyAffect` in [frontend/src/lib/sensing.ts](frontend/src/lib/sensing.ts).
+  - Calibration is now averaged over the first 30 frames (~1s of neutral face) instead of a single-frame snapshot — a brief smile at startup used to lock in a biased baseline. Affect stays null during calibration; gaze/head/gesture/air-writing still flow.
+- [x] **[Core]** Gestures (`THUMBS_UP` / `THUMBS_DOWN` / `POINTING` / `WAVING`) now carry an `opener_hint` via `GESTURE_DIRECTIVES` in [backend/sensing/labels.py](backend/sensing/labels.py). A detected thumbs-up overrides the affect opener and tells the LLM to lead with an affirmation.
+- [x] **[Core]** Air-writing carries a default template bank ([frontend/src/lib/airTemplates.ts](frontend/src/lib/airTemplates.ts): `yes` / `?` / `hi` / `help` / `done` / `more` / `water` / `stop`) — all single-stroke shapes so DTW can match reliably. On match, the word flows through the pipeline three ways: (1) retrieval picks up the word as an extra `PERSONAL` sub-intent with a bucket hint (see `infer_bucket` in [backend/sensing/bucket_keywords.py](backend/sensing/bucket_keywords.py) — e.g. `help` → medical, `water` → daily_routine), (2) the planner includes an explicit "the user air-wrote X — incorporate verbatim if appropriate" instruction in the user message, and (3) the word appears in `logs/turns.jsonl` for debugging. The recognizer has a `MATCH_THRESHOLD` reject gate and `console.debug`s on empty-bank / no-match so unrecognised strokes never reach the backend. To add more templates, append entries to `DEFAULT_AIR_TEMPLATES` as 32-point normalised single-stroke trajectories.
 - [ ] **[Bonus]** Voice + air-writing conflict resolution. Capture short voice (Web Speech API), compare to air-written intent, send a `resolved_intent`
-- [ ] thumbs-up only changes the prompt today — should also boost affirmative candidates in the reranker
+- [ ] Thumbs-up currently biases the opener via the prompt. Once generation emits N candidates, move this to candidate reranking for a stronger signal.
 
 ### Intent decomposition
 
@@ -386,6 +389,7 @@ Heads up: all camera/sensing stuff is in the frontend (MediaPipe JS). Backend ju
 - [ ] **[Core]** API returns one response. Should return multiple candidates so the user can pick (and so the next item works)
 - [ ] **[Core]** Frontend needs a candidate picker — show all the options, let the user click one, send the selection back
 - [ ] **[Bonus]** When user picks a candidate, save the `(query, picked)` pair to a side vector index and check it first next turn
+- [x] LLM temperature bumped from 0.4 → 0.8 in [backend/pipeline/nodes/planner.py](backend/pipeline/nodes/planner.py). The old setting produced near-identical responses across turns even when affect/gesture changed, which made the sensing→output link hard to see. 0.8 gives meaningful lexical variation while staying in the persona's voice.
 
 ### Evals
 
@@ -395,20 +399,21 @@ Live per-turn scores show up in the `EvalPanel`. State:
 |--------|--------|
 | Efficiency | works (SLO check on `t_total`) |
 | Faithfulness | stub, returns 0 |
-| Multimodal alignment | stub, returns 0 |
+| Multimodal alignment | works — affect (sentiment lexicon), gesture (opener regex), gaze (bucket match) |
 | Authenticity | star rating in UI but not saved |
 
 - [ ] **[Eval]** Faithfulness — actually check if the response is grounded in what we retrieved. NLI model, sentence-level. If we didn't retrieve anything, flag `no_evidence` instead of pretending we scored it
 - [ ] **[Eval]** Efficiency — per-turn SLO check is done, but for the writeup we need aggregate latency: p50/p95 across a fixed query set, broken out by LLM tier. Spec target is < 6s
-- [ ] **[Eval]** Multimodal alignment — does the response actually reflect the gesture/affect/gaze? Don't need a model for this, just reuse the word maps the planner already has. Gaze one is trickier — check whether the chunks we ended up using came from the bucket the user was looking at
+- [x] **[Eval]** Multimodal alignment — implemented in `backend/evals/multimodal_alignment.py`. Affect scored by positive/negative lexicon overlap vs. target sentiment, gesture by opener-phrase regex (THUMBS_UP/THUMBS_DOWN/WAVING), gaze by fraction of retrieved chunks matching the looked-at bucket. Returned on every turn as `multimodal_alignment` / `affect_alignment` / `gesture_alignment` / `gaze_alignment`
 - [ ] **[Eval]** Authenticity — the Likert stars are wired up in the UI but go nowhere. Save them, log them with the turn so we can actually look at them later
 - [ ] **[Eval]** For the live in-class eval: figure out the actual session — who rates (partners + experts per spec), how many turns each, what gets shown to them. The Likert form is the easy part; the protocol isn't written down anywhere
 - [ ] **[Eval]** Need an offline version of all three model-driven evals (faithfulness / alignment / efficiency). Aggregate numbers across a fixed query set per persona for the writeup
 
 ### Cleanup
 
-- [ ] move the affect→tone / persona override dicts out of code into a yaml
+- [ ] move the affect → `StyleDirective` config (`_AFFECT_CONFIG` in [intent.py](backend/pipeline/nodes/intent.py)) and the gesture directives ([labels.py](backend/sensing/labels.py)) out of code into a yaml
 - [x] delete `backend/sensing/` (dead code, sensing is in frontend) — done, only `labels.py` remains
+- [x] per-persona affect overrides (`_PERSONA_TONE_OVERRIDES`) deleted — redundant with `stylistic_preferences` in the new persona JSONs
 
 ---
 

backend/evals/multimodal_alignment.py

@@ -1,5 +1,85 @@
-# Multimodal alignment scoring.
-from __future__ import annotations
+import re
+
+_POSITIVE = {
+    "glad",
+    "love",
+    "lucky",
+    "happy",
+    "great",
+    "grateful",
+    "fun",
+    "wonderful",
+    "nice",
+    "amazing",
+    "delighted",
+    "pleased",
+    "yes",
+    "solid",
+}
+_NEGATIVE = {
+    "tired",
+    "hard",
+    "sorry",
+    "unfortunately",
+    "bad",
+    "awful",
+    "regrettably",
+    "difficult",
+    "frustrating",
+    "no",
+    "stop",
+}
+
+_AFFECT_TARGET = {
+    "HAPPY": 1.0,
+    "FRUSTRATED": -0.5,
+    "NEUTRAL": 0.0,
+    "SURPRISED": 0.0,
+}
+
+_GESTURE_OPENER_PATTERNS = {
+    "THUMBS_UP": re.compile(r"^\s*(yes|yeah|totally|for sure|absolutely|sure)\b", re.I),
+    "THUMBS_DOWN": re.compile(r"^\s*(no|nah|not really|i'd rather not)\b", re.I),
+    "WAVING": re.compile(r"^\s*(hi|hey|hello)\b", re.I),
+}
+
+
+def _tokens(text: str) -> set[str]:
+    return set(re.findall(r"\b[a-z]+\b", text.lower()))
+
+
+def _sentiment_score(text: str) -> float:
+    toks = _tokens(text)
+    pos = len(toks & _POSITIVE)
+    neg = len(toks & _NEGATIVE)
+    if pos == 0 and neg == 0:
+        return 0.0
+    return (pos - neg) / (pos + neg)
+
+
+def _affect_alignment(response: str, affect: str | None) -> float:
+    if not affect:
+        return 0.0
+    target = _AFFECT_TARGET.get(affect, 0.0)
+    score = _sentiment_score(response)
+    # distance in [0, 2] → similarity in [0, 1]
+    return max(0.0, 1.0 - abs(score - target) / 2.0)
+
+
+def _gesture_alignment(response: str, gesture_tag: str | None) -> float:
+    if not gesture_tag:
+        return 0.0
+    pattern = _GESTURE_OPENER_PATTERNS.get(gesture_tag)
+    if pattern is None:
+        return 0.5  # gesture has no testable opener; give partial credit
+    return 1.0 if pattern.search(response) else 0.0
+
+
+def _gaze_alignment(chunks: list[dict], gaze_bucket: str | None) -> float:
+    if not gaze_bucket or not chunks:
+        return 0.0
+    matches = sum(1 for c in chunks if c.get("bucket") == gaze_bucket)
+    return matches / len(chunks)
 
 
 def compute_multimodal_alignment(
@@ -9,10 +89,17 @@ def compute_multimodal_alignment(
     gaze_bucket: str | None,
     chunks: list[dict],
 ) -> dict:
-    """Score alignment between non-verbal inputs and generated text."""
+    scores: dict[str, float] = {}
+    if affect:
+        scores["affect_alignment"] = _affect_alignment(response, affect)
+    if gesture_tag:
+        scores["gesture_alignment"] = _gesture_alignment(response, gesture_tag)
+    if gaze_bucket:
+        scores["gaze_alignment"] = _gaze_alignment(chunks, gaze_bucket)
+    overall = sum(scores.values()) / len(scores) if scores else 0.0
     return {
-        "overall_score": 0.0,
-        "affect_alignment": 0.0,
-        "gesture_alignment": 0.0,
-        "gaze_alignment": 0.0,
+        "overall_score": round(overall, 4),
+        "affect_alignment": round(scores.get("affect_alignment", 0.0), 4),
+        "gesture_alignment": round(scores.get("gesture_alignment", 0.0), 4),
+        "gaze_alignment": round(scores.get("gaze_alignment", 0.0), 4),
     }

backend/main.py

@@ -2,6 +2,7 @@
 from __future__ import annotations
 
 import argparse
+import copy
 import json
 import os
 import sys
@@ -10,6 +11,7 @@ import time
 from backend.config.settings import settings
 from backend.guardrails.checks import check_input
 from backend.pipeline.graph import run_pipeline
+from backend.pipeline.nodes.intent import _AFFECT_CONFIG
 from backend.pipeline.state import GenerationConfig, PipelineState
 from backend.retrieval.bucket_priors import uniform_priors
 from backend.retrieval.vector_store import _get_embedder
@@ -49,6 +51,7 @@ def _keyword_intent(query: str) -> tuple[dict, GenerationConfig]:
         else "PERSONAL"
     )
 
+    # `style_constraints` is vestigial — planner reads `generation_config` (below) as the source of truth.
     route = {
         "sub_intents": [
             {
@@ -66,12 +69,8 @@ def _keyword_intent(query: str) -> tuple[dict, GenerationConfig]:
         },
         "affect": "NEUTRAL",
     }
-    gen_config: GenerationConfig = {
-        "max_tokens": settings.max_tokens_neutral,
-        "tone_tag": "[TONE:DEFAULT]",
-        "retrieval_mode": "full",
-        "persona_mod": "baseline",
-    }
+    # Deep-copy: callers may mutate gen_config downstream; never hand them the shared constant.
+    gen_config: GenerationConfig = copy.deepcopy(_AFFECT_CONFIG["NEUTRAL"])
     return route, gen_config
 
 

backend/pipeline/nodes/intent.py

@@ -1,6 +1,7 @@
 # Intent decomposition node — regex-split fragments + BGE zero-shot classifier.
 from __future__ import annotations
 
+import copy
 import re
 import time
 from functools import lru_cache
@@ -88,24 +89,65 @@ _AFFECT_CONFIG: dict[str, GenerationConfig] = {
         "tone_tag": "[TONE:WARM]",
         "retrieval_mode": "full",
         "persona_mod": "amplify_quirks",
+        "style": {
+            "tone_tag": "[TONE:WARM]",
+            "register": "warm, upbeat, affectionate",
+            "prefer_words": [
+                "glad",
+                "love",
+                "lucky",
+                "happy",
+                "great",
+                "grateful",
+                "fun",
+            ],
+            "avoid_words": ["unfortunately", "frankly", "tired", "hard", "sorry"],
+            "opener_hint": None,
+            "exemplar": "Yeah — honestly, that made my week.",
+        },
     },
     "FRUSTRATED": {
         "max_tokens": settings.max_tokens_frustrated,
         "tone_tag": "[TONE:DIRECT_EMPATHETIC]",
         "retrieval_mode": "fast",
         "persona_mod": "suppress_humor",
+        "style": {
+            "tone_tag": "[TONE:DIRECT_EMPATHETIC]",
+            "register": "direct, short, validating — no jokes",
+            "prefer_words": ["okay", "yes", "right", "i hear you", "fair"],
+            "avoid_words": ["hilarious", "ha", "lol", "cheerful", "delightful"],
+            "opener_hint": "Acknowledge the feeling in 3-5 words before the answer.",
+            "exemplar": "Yeah. That's a lot. Short answer: yes.",
+        },
     },
     "NEUTRAL": {
         "max_tokens": settings.max_tokens_neutral,
         "tone_tag": "[TONE:DEFAULT]",
         "retrieval_mode": "full",
         "persona_mod": "baseline",
+        "style": {
+            "tone_tag": "[TONE:DEFAULT]",
+            "register": "natural, conversational",
+            "prefer_words": [],
+            "avoid_words": [],
+            "opener_hint": None,
+            # Empty on purpose — let the persona's own example_phrases carry the register.
+            "exemplar": "",
+        },
     },
     "SURPRISED": {
         "max_tokens": settings.max_tokens_surprised,
         "tone_tag": "[TONE:CLARIFYING]",
         "retrieval_mode": "full",
         "persona_mod": "add_confirmation",
+        "style": {
+            "tone_tag": "[TONE:CLARIFYING]",
+            "register": "curious, clarifying",
+            "prefer_words": ["really", "wait", "huh", "oh"],
+            "avoid_words": [],
+            "opener_hint": "Mirror surprise briefly, then ask a clarifying question.",
+            "exemplar": "Oh — wait, really? Did you mean the Friday one?",
+        },
     },
 }
 
@@ -185,7 +227,8 @@ def run(state: PipelineState) -> dict:
     affect_state = state.get("affect") or {}
     emotion: str = affect_state.get("emotion", "NEUTRAL")
     query: str = state["raw_query"]
-    gen_config = _AFFECT_CONFIG.get(emotion, _AFFECT_CONFIG["NEUTRAL"])
+    # Deep-copy: callers may mutate gen_config downstream; never hand them the shared constant.
+    gen_config = copy.deepcopy(_AFFECT_CONFIG.get(emotion, _AFFECT_CONFIG["NEUTRAL"]))
 
     fragments = _split_query(query)
     priority = "fast" if emotion == "FRUSTRATED" else "normal"

backend/pipeline/nodes/planner.py

@@ -1,30 +1,35 @@
-# Planner node — prompt building, candidate generation, composite ranking.
-from __future__ import annotations
-
 import time
 
 from backend.config.settings import settings
 from backend.generation.llm_client import active_model, chat_complete
 from backend.guardrails.checks import check_output
 from backend.pipeline.intent_kind import classify_intent_kind
-from backend.pipeline.state import PipelineState
-from backend.sensing.labels import GESTURE_TO_TAG
-
-# ── Persona-specific tone tags (applied on top of affect base tag) ─────────────
-
-_PERSONA_TONE_OVERRIDES: dict[str, dict[str, str]] = {
-    "mia_chen": {
-        "HAPPY": "[TONE:WITTY_SARCASTIC]",
-        "FRUSTRATED": "[TONE:DIRECT_EMPATHETIC]",
-    },
-    "gerald_okafor": {
-        "HAPPY": "[TONE:WARM_FORMAL]",
-        "FRUSTRATED": "[TONE:MEASURED_EMPATHETIC]",
-    },
-    "arjun_mehta": {
-        "HAPPY": "[TONE:DIRECT_WARM]",
-        "FRUSTRATED": "[TONE:MINIMAL_DIRECT]",
-    },
+from backend.pipeline.state import PipelineState, StyleDirective
+from backend.sensing.labels import GESTURE_DIRECTIVES
+
+_PERSONA_MOD_INSTRUCTIONS = {
+    "amplify_quirks": "Amplify your characteristic style and personality.",
+    "suppress_humor": "Be direct and supportive. Suppress humor.",
+    "baseline": "Use your natural communication style.",
+    "add_confirmation": "Add a clarifying question or confirmation at the end.",
+    "turnaround": (
+        "Your previous reply missed what you actually meant. Rephrase "
+        "more directly — change the wording meaningfully, not just "
+        "surface tweaks — and end with a one-sentence clarifying "
+        "question to confirm you're on the right track."
+    ),
+    "reverse_stance": (
+        "Your previous reply was substantively wrong — not poorly worded, "
+        "but the wrong content. Take a meaningfully different stance using "
+        "the available memories or, if none fit, honestly say you don't "
+        "know. Do NOT just reword the previous reply."
+    ),
+    "present_state_retry": (
+        "Your previous reply was wrong about your current state. The "
+        "affect signal probably misled you. Either flip the emotional "
+        "read (if you said 'good', try 'not great') or honestly admit "
+        "you're not sure how you feel right now. Do NOT invent details."
+    ),
 }
 
 
@@ -36,22 +41,16 @@ def run_fallback(state: PipelineState) -> dict:
     return _run(state, tier="fallback")
 
 
-# ── Core implementation ────────────────────────────────────────────────────────
-
-
 def _run(state: PipelineState, tier: str) -> dict:
     t0 = time.perf_counter()
 
     profile = state["persona_profile"]
-    user_id = state["user_id"]
     affect = (state.get("affect") or {}).get("emotion", "NEUTRAL")
     gen_cfg = state.get("generation_config") or {}
     chunks = state.get("retrieved_chunks") or []
     history = (state.get("session_history") or [])[-20:]
 
-    tone_tag = _resolve_tone_tag(
-        user_id, affect, gen_cfg.get("tone_tag", "[TONE:DEFAULT]")
-    )
+    style: StyleDirective = gen_cfg["style"]
     gesture_tag = state.get("gesture_tag")
     air_written_text = state.get("air_written_text")
     turnaround_triggered = state.get("turnaround_triggered", False)
@@ -64,7 +63,7 @@ def _run(state: PipelineState, tier: str) -> dict:
         chunks,
         history,
         state["raw_query"],
-        tone_tag,
+        style,
         gen_cfg,
         gesture_tag=gesture_tag,
         air_written_text=air_written_text,
@@ -76,7 +75,7 @@ def _run(state: PipelineState, tier: str) -> dict:
     selected = chat_complete(
         messages=messages,
         max_tokens=gen_cfg.get("max_tokens", settings.max_tokens_neutral),
-        temperature=0.4,
+        temperature=0.8,
         tier=tier,
     )
 
@@ -95,7 +94,7 @@ def _run(state: PipelineState, tier: str) -> dict:
         4,
     )
 
-    augmented_prompt = "\n\n".join(m["content"] for m in messages)
+    augmented_prompt = "\n\n".join(f"[{m['role']}] {m['content']}" for m in messages)
     return {
         "augmented_prompt": augmented_prompt,
         "candidates": [selected],
@@ -107,8 +106,8 @@ def _run(state: PipelineState, tier: str) -> dict:
     }
 
 
-def _resolve_tone_tag(user_id: str, affect: str, default_tag: str) -> str:
-    return _PERSONA_TONE_OVERRIDES.get(user_id, {}).get(affect, default_tag)
+def _format_word_list(words: list[str]) -> str:
+    return ", ".join(words) if words else "(no constraint)"
 
 
 _AFFECT_HINTS = {
@@ -124,7 +123,7 @@ def _build_messages(
     chunks: list[dict],
     history: list[dict],
     query: str,
-    tone_tag: str,
+    style: StyleDirective,
     gen_cfg: dict,
     gesture_tag: str | None = None,
     air_written_text: str | None = None,
@@ -141,7 +140,7 @@ def _build_messages(
         chunks,
         history,
         query,
-        tone_tag,
+        style,
         gen_cfg,
         gesture_tag,
         air_written_text,
@@ -196,37 +195,11 @@ Answering rules:
 --- end character sheet ---"""
 
 
-_PERSONA_MOD_INSTRUCTIONS = {
-    "amplify_quirks": "Amplify your characteristic style and personality.",
-    "suppress_humor": "Be direct and supportive. Suppress humor.",
-    "baseline": "Use your natural communication style.",
-    "add_confirmation": "Add a clarifying question or confirmation at the end.",
-    "turnaround": (
-        "Your previous reply missed what you actually meant. Rephrase "
-        "more directly — change the wording meaningfully, not just "
-        "surface tweaks — and end with a one-sentence clarifying "
-        "question to confirm you're on the right track."
-    ),
-    "reverse_stance": (
-        "Your previous reply was substantively wrong — not poorly worded, "
-        "but the wrong content. Take a meaningfully different stance using "
-        "the available memories or, if none fit, honestly say you don't "
-        "know. Do NOT just reword the previous reply."
-    ),
-    "present_state_retry": (
-        "Your previous reply was wrong about your current state. The "
-        "affect signal probably misled you. Either flip the emotional "
-        "read (if you said 'good', try 'not great') or honestly admit "
-        "you're not sure how you feel right now. Do NOT invent details."
-    ),
-}
-
-
 def _build_user(
     chunks: list[dict],
     history: list[dict],
     query: str,
-    tone_tag: str,
+    style: StyleDirective,
     gen_cfg: dict,
     gesture_tag: str | None,
     air_written_text: str | None,
@@ -262,20 +235,41 @@ def _build_user(
         or "  (start of session)"
     )
 
-    gesture_line = ""
+    merged_opener = style.get("opener_hint")
     if gesture_tag:
-        g_tag = GESTURE_TO_TAG.get(gesture_tag, f"[GESTURE:{gesture_tag}]")
-        gesture_line = f"\nActive gesture signal: {g_tag}"
+        directive = GESTURE_DIRECTIVES.get(gesture_tag)
+        if directive:
+            # Gesture opener wins over affect opener — a deliberate thumbs-up is a stronger signal than inferred affect.
+            merged_opener = directive["opener_hint"]
 
-    air_writing_line = ""
+    air_writing_block = ""
     if air_written_text:
-        air_writing_line = f'\nThe user air-wrote: "{air_written_text}" — treat as supplementary intent.'
+        air_writing_block = (
+            f'\nThe user air-wrote: "{air_written_text}". '
+            "If this looks like a name, noun, or short phrase, "
+            "incorporate it verbatim into your response; "
+            "otherwise use it as a hint about what they're trying to say."
+        )
 
-    persona_instruction = _PERSONA_MOD_INSTRUCTIONS.get(
-        gen_cfg.get("persona_mod", "baseline"),
-        _PERSONA_MOD_INSTRUCTIONS["baseline"],
+    persona_mod = gen_cfg.get("persona_mod", "baseline")
+    persona_instruction_line = (
+        f"\n{_PERSONA_MOD_INSTRUCTIONS[persona_mod]}"
+        if persona_mod in _PERSONA_MOD_INSTRUCTIONS and persona_mod != "baseline"
+        else ""
     )
 
+    directive_lines = [
+        f"- Register: {style['register']}",
+        f"- Prefer words like: {_format_word_list(style['prefer_words'])}",
+        f"- Avoid words like: {_format_word_list(style['avoid_words'])}",
+        f"- Opener: {merged_opener or 'no constraint'}",
+    ]
+    if style.get("exemplar"):
+        directive_lines.append(
+            f'- In this register, a sentence sounds like: "{style["exemplar"]}"'
+        )
+    directive_block = "Style directive:\n" + "\n".join(directive_lines)
+
     turnaround_line = ""
     if rejected_response:
         safe_rejected = rejected_response.replace('"', "'").replace("\n", " ")[:300]
@@ -287,8 +281,7 @@ def _build_user(
     if intent_kind == "present_state":
         affect_hint = _AFFECT_HINTS.get(affect, _AFFECT_HINTS["NEUTRAL"])
         return f"""\
-{tone_tag}{gesture_line}{air_writing_line}{turnaround_line}
-{persona_instruction}
+{directive_block}{air_writing_block}{turnaround_line}{persona_instruction_line}
 
 The partner is asking about your present state (right now, today).
 Your autobiographical memories do NOT contain this — do not fabricate details from them.
@@ -307,8 +300,7 @@ Reply as {persona_name} in 1–2 sentences, first person.
 - Do NOT use autobiographical facts (job, family, hobbies) unless the partner asked."""
 
     return f"""\
-{tone_tag}{gesture_line}{air_writing_line}{turnaround_line}
-{persona_instruction}
+{directive_block}{air_writing_block}{turnaround_line}{persona_instruction_line}
 
 Personal memories:
 {memory_block}

backend/pipeline/state.py

@@ -43,14 +43,25 @@ class IntentRoute(TypedDict):
     affect: str
 
 
+class StyleDirective(TypedDict):
+    tone_tag: str  # e.g. "[TONE:WARM]" — kept for logging + eval
+    register: str  # short register phrase, e.g. "warm, upbeat, affectionate"
+    prefer_words: list[str]  # lexical bias — words to steer toward
+    avoid_words: list[str]  # anti-patterns — words to steer away from
+    opener_hint: str | None  # structural hint for the opening clause
+    exemplar: str  # one short sentence in the target register
+
+
 class GenerationConfig(TypedDict):
     max_tokens: int
-    tone_tag: str  # e.g. "[TONE:WITTY_SARCASTIC]"
+    tone_tag: str  # legacy tag (kept in sync with style["tone_tag"] for existing log consumers)
     retrieval_mode: str  # "fast" | "full"
     persona_mod: str
     # persona_mod values:
     #   "amplify_quirks" | "suppress_humor" | "baseline"
     #   | "add_confirmation" | "turnaround"
+    #   | "reverse_stance" | "present_state_retry"
+    style: StyleDirective
 
 
 class LatencyLog(TypedDict):

backend/sensing/bucket_keywords.py

@@ -1,9 +1,26 @@
 _BUCKET_KEYWORDS: list[tuple[str, tuple[str, ...]]] = [
-    ("medical", ("medication", "medicine", "doctor", "health", "allergic", "therapy")),
+    # AAC air-writing templates (help/water/stop/done/more) are mapped here too —
+    # when a partner/user signals one of these, retrieval pulls from the matching bucket.
+    (
+        "medical",
+        (
+            "medication",
+            "medicine",
+            "doctor",
+            "health",
+            "allergic",
+            "therapy",
+            "help",
+            "stop",
+        ),
+    ),
     ("family", ("family", "mom", "dad", "brother", "sister", "parents")),
     ("hobbies", ("hobby", "like to do", "enjoy", "weekend", "fun")),
-    ("daily_routine", ("routine", "morning", "wake", "sleep", "daily")),
-    ("social", ("friend", "social", "people", "party", "community")),
+    (
+        "daily_routine",
+        ("routine", "morning", "wake", "sleep", "daily", "water", "done", "more"),
+    ),
+    ("social", ("friend", "social", "people", "party", "community", "hi")),
 ]
 
 

backend/sensing/labels.py

@@ -1,6 +1,18 @@
-GESTURE_TO_TAG: dict[str, str] = {
-    "THUMBS_UP": "[GESTURE:THUMBS_UP][TONE:AFFIRMATIVE]",
-    "THUMBS_DOWN": "[GESTURE:THUMBS_DOWN][TONE:NEGATIVE]",
-    "POINTING": "[GESTURE:POINTING][INTENT:REFERENTIAL]",
-    "WAVING": "[GESTURE:WAVING][INTENT:GREETING]",
+GESTURE_DIRECTIVES: dict[str, dict[str, str]] = {
+    "THUMBS_UP": {
+        "tone": "[GESTURE:THUMBS_UP][TONE:AFFIRMATIVE]",
+        "opener_hint": "Open with an affirmation (Yes / Totally / For sure).",
+    },
+    "THUMBS_DOWN": {
+        "tone": "[GESTURE:THUMBS_DOWN][TONE:NEGATIVE]",
+        "opener_hint": "Open by declining or disagreeing briefly.",
+    },
+    "POINTING": {
+        "tone": "[GESTURE:POINTING][INTENT:REFERENTIAL]",
+        "opener_hint": "Treat the query as referring to a specific named thing.",
+    },
+    "WAVING": {
+        "tone": "[GESTURE:WAVING][INTENT:GREETING]",
+        "opener_hint": "Open with a greeting.",
+    },
 }

frontend/src/hooks/useSensing.ts

@@ -13,6 +13,7 @@ import {
   AirWriter,
   HeadPoseTracker,
 } from "../lib/sensing";
+import { DEFAULT_AIR_TEMPLATES } from "../lib/airTemplates";
 
 const EMA_ALPHA = 0.3;
 
@@ -20,11 +21,12 @@ export function useSensing() {
   const faceLandmarkerRef = useRef<FaceLandmarker | null>(null);
   const handLandmarkerRef = useRef<HandLandmarker | null>(null);
   const gazeTrackerRef = useRef(new GazeTracker());
-  const airWriterRef = useRef(new AirWriter());
+  const airWriterRef = useRef(new AirWriter(DEFAULT_AIR_TEMPLATES));
   const headTrackerRef = useRef(new HeadPoseTracker());
   const calibratePendingRef = useRef(false);
   const headDebugRef = useRef({ dx: 0, dy: 0, maxAbsDx: 0, maxAbsDy: 0, crossings: 0 });
   const neutralLCPRef = useRef<number | null>(null);
+  const calibBufferRef = useRef<number[]>([]);
   const smoothedRef = useRef({ MAR: 0, EAR: 0.3, BRI: -0.3, LCP: 0 });
   const initingRef = useRef(false);
   const [ready, setReady] = useState(false);
@@ -108,9 +110,18 @@ export function useSensing() {
       if (faceResult.faceLandmarks && faceResult.faceLandmarks.length > 0) {
         const landmarks = faceResult.faceLandmarks[0];
 
+        // Average the raw LCP (vertical corner pull, pre-offset) over ~30 frames
+        // of the user's face before locking neutral. Single-frame calibration is
+        // too noisy and tended to bake in a momentary smile as "neutral".
+        // During calibration, affect stays null but gaze/head/gesture still flow.
         if (neutralLCPRef.current === null) {
-          neutralLCPRef.current =
-            (landmarks[61].x + landmarks[291].x) / 2;
+          const raw0 = computeAffectVector(landmarks, 0);
+          calibBufferRef.current.push(raw0.LCP);
+          if (calibBufferRef.current.length >= 30) {
+            const sum = calibBufferRef.current.reduce((a, b) => a + b, 0);
+            neutralLCPRef.current = sum / calibBufferRef.current.length;
+            calibBufferRef.current = [];
+          }
         }
 
         if (calibratePendingRef.current) {
@@ -118,18 +129,21 @@ export function useSensing() {
           calibratePendingRef.current = false;
         }
 
-        const raw = computeAffectVector(landmarks, neutralLCPRef.current);
+        if (neutralLCPRef.current !== null) {
+          const raw = computeAffectVector(landmarks, neutralLCPRef.current);
 
-        const prev = smoothedRef.current;
-        const smoothed = {
-          MAR: EMA_ALPHA * raw.MAR + (1 - EMA_ALPHA) * prev.MAR,
-          EAR: EMA_ALPHA * raw.EAR + (1 - EMA_ALPHA) * prev.EAR,
-          BRI: EMA_ALPHA * raw.BRI + (1 - EMA_ALPHA) * prev.BRI,
-          LCP: EMA_ALPHA * raw.LCP + (1 - EMA_ALPHA) * prev.LCP,
-        };
-        smoothedRef.current = smoothed;
+          const prev = smoothedRef.current;
+          const smoothed = {
+            MAR: EMA_ALPHA * raw.MAR + (1 - EMA_ALPHA) * prev.MAR,
+            EAR: EMA_ALPHA * raw.EAR + (1 - EMA_ALPHA) * prev.EAR,
+            BRI: EMA_ALPHA * raw.BRI + (1 - EMA_ALPHA) * prev.BRI,
+            LCP: EMA_ALPHA * raw.LCP + (1 - EMA_ALPHA) * prev.LCP,
+          };
+          smoothedRef.current = smoothed;
+
+          affect = classifyAffect(smoothed);
+        }
 
-        affect = classifyAffect(smoothed);
         gazeBucket = gazeTrackerRef.current.process(landmarks);
         headSignal = headTrackerRef.current.process(landmarks);
         headDebugRef.current = headTrackerRef.current.debug;
@@ -182,6 +196,7 @@ export function useSensing() {
 
   const resetCalibration = useCallback(() => {
     neutralLCPRef.current = null;
+    calibBufferRef.current = [];
     smoothedRef.current = { MAR: 0, EAR: 0.3, BRI: -0.3, LCP: 0 };
     gazeTrackerRef.current.reset();
     headTrackerRef.current.reset();

frontend/src/lib/airTemplates.ts

@@ -0,0 +1,108 @@
+// Default air-writing template bank.
+// Each template is a normalised 32-point [x, y] trajectory (coords in [0, 1]).
+// Matched against live trajectories via DTW in AirWriter.recognise.
+// To add a new template: pick a distinctive *single-stroke* shape,
+// sample ~32 evenly-spaced points from stroke start → end, normalise
+// x/y into [0, 1], and add an entry to DEFAULT_AIR_TEMPLATES.
+//
+// DTW quality tips:
+// - Stick to single-stroke shapes. Multi-stroke shapes (like an X) look
+//   like a teleport to DTW and will mis-match.
+// - Shapes should be distinctive in direction and extent — a small
+//   check-mark and a big slash look similar after normalisation.
+
+function linear(from: [number, number], to: [number, number], n: number): [number, number][] {
+  const out: [number, number][] = [];
+  for (let i = 0; i < n; i++) {
+    const t = i / (n - 1);
+    out.push([from[0] + t * (to[0] - from[0]), from[1] + t * (to[1] - from[1])]);
+  }
+  return out;
+}
+
+function concat(...segs: [number, number][][]): [number, number][] {
+  const out: [number, number][] = [];
+  for (const s of segs) out.push(...s);
+  return resample(out, 32);
+}
+
+function resample(pts: [number, number][], n: number): [number, number][] {
+  if (pts.length < 2) return pts;
+  const out: [number, number][] = [];
+  for (let i = 0; i < n; i++) {
+    const t = (i / (n - 1)) * (pts.length - 1);
+    const lo = Math.floor(t);
+    const hi = Math.min(lo + 1, pts.length - 1);
+    const frac = t - lo;
+    out.push([
+      pts[lo][0] + frac * (pts[hi][0] - pts[lo][0]),
+      pts[lo][1] + frac * (pts[hi][1] - pts[lo][1]),
+    ]);
+  }
+  return out;
+}
+
+// check-mark: short down-right, then long up-right → affirmation
+const YES: [number, number][] = concat(
+  linear([0.0, 0.5], [0.35, 1.0], 12),
+  linear([0.35, 1.0], [1.0, 0.0], 20)
+);
+
+// question-mark: curve over the top, then down to the dot → clarifying
+const QUESTION: [number, number][] = concat(
+  linear([0.1, 0.25], [0.5, 0.0], 8),
+  linear([0.5, 0.0], [0.9, 0.25], 8),
+  linear([0.9, 0.25], [0.5, 0.55], 8),
+  linear([0.5, 0.55], [0.5, 1.0], 8)
+);
+
+// zig-zag wave across the top → greeting
+const HI: [number, number][] = concat(
+  linear([0.0, 0.0], [0.25, 1.0], 8),
+  linear([0.25, 1.0], [0.5, 0.0], 8),
+  linear([0.5, 0.0], [0.75, 1.0], 8),
+  linear([0.75, 1.0], [1.0, 0.0], 8)
+);
+
+// straight vertical line bottom→top → "help" (raise hand / SOS mental model)
+const HELP: [number, number][] = linear([0.5, 1.0], [0.5, 0.0], 32);
+
+// horizontal line left→right → "done" (close / finish)
+const DONE: [number, number][] = linear([0.0, 0.5], [1.0, 0.5], 32);
+
+// plus-sign-ish as a single stroke: long down, backtrack up, then across → "more"
+// mimics drawing "+"  as one continuous stroke (down, back, right)
+const MORE: [number, number][] = concat(
+  linear([0.5, 0.0], [0.5, 1.0], 12),
+  linear([0.5, 1.0], [0.5, 0.5], 6),
+  linear([0.5, 0.5], [1.0, 0.5], 14)
+);
+
+// single wave (down-up-down-up smooth) → "water" (fluid/ocean mental model)
+const WATER: [number, number][] = concat(
+  linear([0.0, 0.5], [0.2, 0.9], 6),
+  linear([0.2, 0.9], [0.4, 0.1], 8),
+  linear([0.4, 0.1], [0.6, 0.9], 8),
+  linear([0.6, 0.9], [0.8, 0.1], 8),
+  linear([0.8, 0.1], [1.0, 0.5], 2)
+);
+
+// square/box (traced as one stroke) → "stop"
+// start top-left, go right, down, left, up — closing the box
+const STOP: [number, number][] = concat(
+  linear([0.0, 0.0], [1.0, 0.0], 8),
+  linear([1.0, 0.0], [1.0, 1.0], 8),
+  linear([1.0, 1.0], [0.0, 1.0], 8),
+  linear([0.0, 1.0], [0.0, 0.0], 8)
+);
+
+export const DEFAULT_AIR_TEMPLATES: Map<string, [number, number][]> = new Map([
+  ["yes", YES],
+  ["?", QUESTION],
+  ["hi", HI],
+  ["help", HELP],
+  ["done", DONE],
+  ["more", MORE],
+  ["water", WATER],
+  ["stop", STOP],
+]);

frontend/src/lib/sensing.ts

@@ -11,12 +11,16 @@ interface AffectVector {
 
 export function classifyAffect(v: AffectVector): Affect {
   // BRI is relative (browMid.y - eyeCenter.y) / interOcular — more negative = brows raised higher
-  // LCP is relative to calibrated neutral — positive = corners pulled up (smile)
+  // LCP is vertical offset of lip corners from mouth center, normalised by inter-ocular,
+  //   relative to calibrated neutral — positive = corners pulled UP (smile), negative = DOWN (frown)
   // MAR is absolute ratio — higher = mouth more open
-  // EAR is absolute ratio — lower = eyes more closed
+  // EAR is absolute ratio — lower = eyes more closed / squinting
   if (v.BRI < -0.35 && v.MAR > 0.4) return "SURPRISED";
-  if (v.EAR < 0.12 && v.LCP < -0.005) return "FRUSTRATED";
-  if (v.LCP > 0.005) return "HAPPY";
+  // FRUSTRATED: a clear frown, OR brows lowered + squinting — either signals displeasure
+  if (v.LCP < -0.015) return "FRUSTRATED";
+  if (v.BRI > -0.2 && v.EAR < 0.18) return "FRUSTRATED";
+  // HAPPY: meaningful upward pull of lip corners (tighter than the old 0.005)
+  if (v.LCP > 0.015) return "HAPPY";
   return "NEUTRAL";
 }
 
@@ -55,8 +59,14 @@ export function computeAffectVector(
   // Raising brows moves them toward y=0, making this value more negative.
   const BRI = (browMid.y - eyeCenter.y) / (interOcular + 1e-6);
 
-  const LCP =
-    (landmarks[CORNER_LEFT].x + landmarks[CORNER_RIGHT].x) / 2 - neutralLCP;
+  // Lip-corner pull: average y of the two corners vs. mouth vertical centre,
+  // normalised by inter-ocular distance, relative to calibrated neutral.
+  // MediaPipe y increases downward, so corners rising above the mouth centre → negative raw,
+  // which we flip so smile = positive. Subtracting the calibrated neutral removes per-face bias.
+  const mouthCentreY = (landmarks[MOUTH_TOP].y + landmarks[MOUTH_BOTTOM].y) / 2;
+  const cornerAvgY = (landmarks[CORNER_LEFT].y + landmarks[CORNER_RIGHT].y) / 2;
+  const rawLCP = (mouthCentreY - cornerAvgY) / (interOcular + 1e-6);
+  const LCP = rawLCP - neutralLCP;
 
   return { MAR, EAR, BRI, LCP };
 }
@@ -524,7 +534,13 @@ export class AirWriter {
   }
 
   private recognise(trajectory: [number, number][]): string | null {
-    if (trajectory.length < 5 || this.templates.size === 0) return null;
+    if (trajectory.length < 5) {
+      return null;
+    }
+    if (this.templates.size === 0) {
+      console.debug("[AirWriter] stroke completed but template bank is empty");
+      return null;
+    }
     const query = normaliseTrajectory(trajectory);
     let bestChar: string | null = null;
     let bestDist = Infinity;
@@ -535,6 +551,15 @@ export class AirWriter {
         bestChar = char;
       }
     }
+    // Reject poor matches so we don't pass garbage to the LLM.
+    // Threshold is empirical — tune once real users test this.
+    const MATCH_THRESHOLD = 8.0;
+    if (bestDist > MATCH_THRESHOLD) {
+      console.debug(
+        `[AirWriter] no template matched (best='${bestChar}', dist=${bestDist.toFixed(2)})`
+      );
+      return null;
+    }
     return bestChar;
   }