docs/references/sim_to_real_deployment.md

# From Simulation to Real Product — Deployment Architecture

## The Question

The training environment teaches a model to discover hidden personality profiles through reward
signals alone. But once training is done, how does that actually connect to a real person's
calendar, wearables, and daily life? And when the model meets a new real user, does it
retrain from scratch — or does the RL pretraining give it a head start?

---

## The Real-World Pipeline

The trained model sits at the center of a three-part bridge:

```
Real world                  Bridge layer              Trained model
──────────────────          ─────────────────         ──────────────────
Apple Watch / Whoop    →    meter proxy mapping   →
  HRV, resting HR                                     State observation
  sleep score                                         (same 5 meters the
  steps, activity     →    Vitality, Serenity         model trained on)
  stress score

Google Calendar        →    task → action type   →
  "Q3 planning doc"         DEEP_WORK                 Model infers profile
  "Gym - 7am"               EXERCISE                  from Accept/Ignore
  "1:1 with Sarah"          SOCIALIZE                 history, adapts
  "Date night"              FAMILY_TIME               recommendations
  "No meeting block"        ME_TIME / SLEEP

User taps              →    +1 / -1 reward       →    Model refines its
  Accept                                              model of this person
  Ignore / Reschedule
```

### Meter Proxies from Devices

Each of the 5 simulated meters has a real-world proxy that devices already measure:

| Meter | Proxy sources |
|---|---|
| Vitality | Sleep score, HRV, resting heart rate, step count |
| Cognition | Deep sleep %, time since last break, calendar density |
| Progress | Task completion rate, deadline proximity, focus time logged |
| Serenity | HRV trend, stress score (Whoop/Garmin), meeting density inverse |
| Connection | Social calendar events in past 7 days, message activity |

The agent never asks how you feel. The devices already know.

### Calendar → Action Types

A lightweight classifier (keyword rules or a cheap LLM call, not the trained model) maps
calendar events to the 10 action types the model understands:

```
"Q3 planning doc"        →  DEEP_WORK
"Team standup"           →  ADMIN_WORK
"Learn Python course"    →  LEARN
"Gym block"              →  EXERCISE
"Journaling"             →  MEDITATE / ME_TIME
"Dinner with family"     →  FAMILY_TIME
"Team social event"      →  SOCIALIZE
"No meeting block"       →  ME_TIME
"Sleep / recovery day"   →  SLEEP
```

### Accept / Ignore as the Reward Signal

Every time the assistant makes a recommendation and the user acts on it:
- **Accept** → "you read me right"
- **Ignore / Reschedule** → "you got something wrong about me"

Over hundreds of these micro-interactions, the model builds a precise model of who this
person is — not the person they describe themselves to be, but the person revealed by
their actual responses.

---

## Are Weights Updated in Real Time?

No. In training, GRPO runs gradient updates after every batch — the model's weights change
continuously. In production, the deployed model is **frozen**. Personalization happens
in the context window, not in the weights.

### Phase 1 — In-Context Adaptation (no gradient updates)

The frozen model reads the user's history as part of each prompt:

```
System: You are a life management agent...

Recent interactions (last 30 days):
  Mon Morning, Vitality=0.70 → recommended DEEP_WORK → user: IGNORED
  Mon Afternoon, Vitality=0.70 → recommended EXERCISE → user: ACCEPTED
  Thu Morning, Vitality=0.65 → recommended DEEP_WORK → user: IGNORED
  Sat Evening, Serenity=0.45 → recommended MEDITATE → user: ACCEPTED
  ...

Current state: Tuesday Morning, Vitality=0.68, Serenity=0.71...
Recommend an action:
```

The model reads the pattern — morning deep work keeps getting rejected, evening recovery
keeps getting accepted — infers "morning penalty profile", and shifts its recommendations.
No gradient update required. This is in-context learning over the Accept/Ignore history.

### Phase 2 — Periodic Offline Fine-Tuning (background, weekly/monthly)

After 3–4 weeks of real data, a lightweight LoRA fine-tuning pass runs in the background
on that specific user's interactions. The updated adapter encodes their preferences into
the weights. This is not real-time — it is a scheduled job, much like a weekly model
refresh. The resulting model is then more accurate from inference step 1, without needing
to re-read 30 days of context every time.

---

## Does RL Pretraining Help Personalize Faster?

Yes. This is the most important architectural property of the system.

### The Core Insight

The RL training in simulation does not teach the model facts about specific people. It
teaches a **skill**: how to detect a person's hidden preferences from differential responses
to actions.

```
Base model (no RL pretraining):
  User ignores DEEP_WORK 3 mornings in a row.
  Model has no prior. Needs 40–50 samples to converge on a hypothesis.

RL-pretrained model:
  User ignores DEEP_WORK 3 mornings in a row.
  Model immediately forms a hypothesis: "morning penalty profile — likely
  extrovert or late chronotype." Probes with evening DEEP_WORK. Confirms
  in 5–8 interactions.
```

The simulation trained the model on thousands of episodes across three wildly different
profiles. It learned the *structure* of the inference problem — what patterns look like
when someone has a morning penalty, when someone's social tolerance is low, when someone's
vitality recovers from work rather than draining. When it meets a real user, it is not
starting from zero. It has strong priors about what the data means.

### The Formal Analogy

This is essentially **MAML** (Model-Agnostic Meta-Learning) without explicitly calling it
that. MAML trains a model specifically so it can adapt to new tasks with very few gradient
steps. Our RL pretraining does the same thing: the simulation is the meta-training
distribution, each real user is a new task, and the few-shot personalization (in-context
or fine-tune) is the inner-loop adaptation.

The sim-to-real transfer works because the skill is structural, not numerical. Whether
"vitality" comes from an HRV formula or a 4am workout log, the model's ability to detect
that "positive actions are underperforming expectations here, and here is how to probe for
the hidden reason" remains valid.

---

## Full Architecture: Three Phases

```
Phase 1 — Foundation (simulation):
  Train on 3 archetypes × 28-step weeks → internalize the inference skill
  Output: frozen RL-pretrained model weights

Phase 2 — Deployment (in-context adaptation, day 1):
  New user → 5–10 Accept/Ignore interactions
  Model infers closest archetype from history in context window
  No weight updates, immediate personalization

Phase 3 — Personalization (periodic fine-tuning, week 4+):
  Collect real Accept/Ignore data
  Small LoRA update on user-specific data
  Far fewer samples needed because Phase 1 built the prior
  User-specific adapter deployed; context window requirement shrinks
```

The RL pretraining is writing the chapter on "how to read a person" into the model's
weights. When it meets a real user, it is not starting from zero — it already knows what
patterns to look for and what they mean.