lessons.md

Lessons

Stuff that didn't work and what we did about it. Roughly in the order it happened, because most of the bad decisions came from not understanding the last one.

The LLM intent router that ate 114 seconds

First version of intent decomposition was an LLM call — Gemma4:31b-cloud, Pydantic-validated JSON, 3 retries on schema failure with the validation error prepended. Looked clean on paper.

In practice, Gemma4 would sometimes emit JSON that hit max_tokens=512 mid-object. Truncated. Validator failed. Retry. Same truncation. Retry again. By the time the hard fallback kicked in, ~114 seconds had gone by with nothing on screen except a spinner.

The retries were the problem. We'd treated them like a cheap safety net, but each one was a 30+ second round-trip. And when the first attempt fails with a specific failure mode (truncation), the retry almost always hits the same wall. One clean try with a sensible fallback beats three slow tries that all die the same way.

We briefly eyed response_format=json_schema as a fix but Ollama Cloud doesn't expose it yet.

So we ripped out the LLM. Then we broke multi-intent.

Second pass: kill the LLM, use keyword matching on the whole query. Fast, deterministic, done.

Except "how are you and what is the capital of France?" now became a single PERSONAL sub-intent. The whole point of decomposition — routing the two halves to different pools — was gone. We'd deleted the feature to make it fast.

Speed isn't the only axis. "Agentic" here means splitting the query into typed sub-queries and routing them; not just "an LLM is involved somewhere." A faster solution that doesn't do what the spec asks is not a solution.

What actually worked: split + zero-shot BGE

Regex-split the query on and / but / punctuation, classify each fragment via cosine similarity against 5 seed sentences per class using the BGE embedder we already had loaded for retrieval.

No LLM, no retries, no new dependencies. Median latency went from 114s to ~30ms on the same input. All three intents routed correctly.

Moral of the story: the cheapest classifier that works is almost always the right one for a prototype. We were already paying for BGE; using it for classification too cost us nothing.

The classifier over-matched CONTEXTUAL

Live test, turn 11 of a Forrest Gump session. Partner: "give me a detailed introduction." Classifier: CONTEXTUAL. Retrieval searched session history, found three weak matches, grounded the LLM prompt in basically nothing about Forrest. The LLM flailed, guardrail caught something, user got "I don't know."

Problem was that CONTEXTUAL exemplars like "what were we talking about" cast too wide a net — any meta-shaped question slid into that bucket. A single threshold (> 0.35) didn't guard against it.

Fix used two extra signals: CONTEXTUAL has to beat the runner-up by a margin (0.08), and the fragment has to contain an actual discourse word (earlier, mentioned, just, repeat, said) matched at word boundaries. Low-confidence goes to PERSONAL, not OPEN_DOMAIN — safer fallback for a persona bot.

One-dimensional thresholds are a weak guard. Adding a margin signal and a structural word cue made wrong classifications much harder without changing what the happy path does.

CONTEXTUAL was fighting personal grounding

Originally we had CONTEXTUAL as a replacement for PERSONAL — "this turn is about what we just said, so search history instead of memories." Wrong. Even when the user asks about prior conversation, the response still needs to sound like the persona. Session history is extra context, not a source of truth.

Now CONTEXTUAL always pulls persona memory first, then layers on relevant history (score ≥ 0.5). Never an empty personal prompt.

Think about where the LLM's source of truth is. For a persona bot, that's the persona's memories, every time. Other signals go on top.

Gemma4 started writing the character brief as output

Early testing, THINKING_MODE=off. Partner: "hi". LLM:

The user wants me to roleplay as Abed Nadir from Community.
Key characteristics:
- Autism spectrum (canonically coded, not explicitly diagnosed)
- Occasional selective mutism during sensory overload
...

It was writing the brief, not being the character. Our prompt front-loaded Abed's condition and voice, then asked for a response at the bottom. Gemma4 treated the whole thing as a writing assignment — summarize first, respond second. We'd accidentally written a creative writing prompt.

Two fixes stacked:

1. Anti-meta rules at the top and bottom of the prompt: "never narrate, analyze, describe, or list traits. Never say 'As an AI', 'The user wants me to', 'Key characteristics'..." Models weight the start and end of a prompt more than the middle; saying the rule in both spots is cheap.
2. THINKING_MODE=suppress in .env. Ollama supports a /no_think prefix on the user message; this turns it on. Gemma4 stopped emitting the scratchpad entirely.

Instruction-tuned models will follow whatever instruction looks most like the task. If your prompt looks like a character brief, the model may complete the brief. State "do not narrate" explicitly, and use the no-think flag when the model supports it.

The guardrail saved us once

After the prompt and /no_think fixes, a test run still leaked — "The user wants me to roleplay as Raymond..." came back from the LLM. But the user never saw it. The output guardrail caught the phrase and swapped in the safe fallback.

Belt-and-braces output checks are worth the effort. When the prompt was wrong, the guardrail was still right.

Open-domain tempted us to build a web search

First instinct when we added OPEN_DOMAIN was "great, now we need a web search adapter." But the product isn't a search engine — it's an AAC user's voice. If someone asks Mia for the capital of France, the answer is "Paris" in her voice. The LLM already knows basic facts; Mia's persona is the scarce thing. Piping in a Wikipedia snippet would dilute her voice, not enrich it.

So OPEN_DOMAIN just emits a stub chunk that tells the LLM to answer from its own knowledge. Cheap, aligned with the product, one less thing to break.

When you see a retrieval-shaped problem, don't assume a retriever is the right answer.

Caching contextual embeddings was a waste of thought

At one point we worried about retrieve_from_history re-encoding the session window every turn. Measured it: 43ms even with 80 turns of history. The LLM call was taking 1.5–95 seconds. Shaving 30ms off a 20-second turn is 0.1%, invisible.

Measure before you optimize, even when the waste seems obvious.

Monolithic prompts don't cache

Our planner built one giant user message with the character sheet (stable per persona) and the retrieved chunks (different every turn) mashed together. Prompt caches match prefixes exactly — one byte change in the retrieval block invalidates the whole prompt, including ~300 tokens of character sheet that hadn't changed.

Split it: system message holds the stable character sheet and answering rules, user message holds the per-turn retrieval and query. Provider caches the system prefix → every turn after the first skips prefill on ~300 tokens.

Structure matters as much as content once you care about latency. Stable stuff goes in the system message, per-turn stuff goes in the user message. Costs nothing in capability, compounds across turns.

---

Principles we kept circling back to

Measure first. Every good decision here was triggered by a number — 114s, 43ms, 30ms. Every bad decision was triggered by a hunch.

Three pools because we have three sources. Not four, not five. A category without a real retriever behind it just confuses the classifier.

Short prompts behave better. Every time we trimmed something, the model was more consistent.

Every path produces at least one chunk. Empty retrieval blocks were the fastest route to a hallucination. CONTEXTUAL with no history, OPEN_DOMAIN with nothing wired up, a classifier returning an empty sub-intent list — all have fallbacks now.