Create v4_1-handoff.md

(docs): v4.1 handoff improvement and changes to be applied in the v4 run

docs/v4_1-handoff.md

@@ -0,0 +1,280 @@
+# V4.1 / V4.2 Handoff
+
+**Date:** 2026-04-27  
+**Context:** V4 completed 200 steps, eval_best=0.476 at step 130. Training plateaued due to
+LR decay and data starvation (13.5% of one epoch seen). Model learned (+33% over SFT baseline).
+Recipe is validated. V4.1 squeezes the 0.5B before scaling to 3.7B.
+
+---
+
+## V4.1 — Four Changes Only
+
+### Change 1: Fix the JSON Parser
+
+**What:** Replace the regex-based `_extract_json()` in the reward functions with a robust
+parser that handles Portuguese decimal commas and LLM formatting quirks.
+
+**Why:** The model is writing `"sentiment_score": 4,5` (correct PT-BR format). Your parser
+calls `json.loads()`, which fails, and scores the completion near-zero. The model is being
+penalized for correct behavior. This is reward misspecification — the single highest-ROI fix
+available at zero training cost.
+
+**Do this before launching any training run.** Run it against `data/pairs/eval.jsonl` first
+and measure how many previously-failing extractions now parse correctly.
+
+```python
+import json, re
+
+def _normalize_pt_decimals(s: str) -> str:
+    """Convert PT-BR decimals (4,5) to JSON-valid (4.5), only outside quoted strings."""
+    result, in_string, escape_next = [], False, False
+    i = 0
+    while i < len(s):
+        c = s[i]
+        if escape_next:
+            result.append(c); escape_next = False; i += 1; continue
+        if c == '\\' and in_string:
+            result.append(c); escape_next = True; i += 1; continue
+        if c == '"':
+            in_string = not in_string; result.append(c); i += 1; continue
+        if not in_string:
+            m = re.match(r'(\d+),(\d+)', s[i:])
+            if m:
+                result.append(m.group(1) + '.' + m.group(2))
+                i += len(m.group(0)); continue
+        result.append(c); i += 1
+    return ''.join(result)
+
+def _extract_json(text: str) -> dict | None:
+    stripped = re.sub(r'^```(?:json)?\s*|\s*```$', '', text.strip(), flags=re.MULTILINE).strip()
+    for attempt in [stripped, _normalize_pt_decimals(stripped)]:
+        try:
+            result = json.loads(attempt)
+            if isinstance(result, dict):
+                return result
+        except (json.JSONDecodeError, TypeError):
+            pass
+        # Try finding first {...} block
+        match = re.search(r'\{[\s\S]*\}', attempt)
+        if match:
+            try:
+                result = json.loads(match.group())
+                if isinstance(result, dict):
+                    return result
+            except (json.JSONDecodeError, TypeError):
+                pass
+    return None
+```
+
+---
+
+### Change 2: Triple the Training Steps
+
+**What:** `MAX_STEPS = 200 → 600`
+
+**Why:** V4 trained on 13.5% of available data. Of ~1,480 training prompts, only ~400 were
+sampled. The insights (148 prompts) and push (148 prompts) tasks may have contributed fewer
+than 50 training examples each — not enough for meaningful policy updates. 600 steps at
+2 prompts/step = 1,200 unique samples, covering ~80% of the training set once.
+
+No other changes needed. Just increase the step count.
+
+---
+
+### Change 3: Replace LR Schedule
+
+**What:** `lr_scheduler_type = "cosine" → "constant_with_warmup"`
+
+**Why:** The V4 cosine schedule decayed to ~1.5×10⁻¹⁰ by step 200. The model spent the
+last 70 steps (130→200) with an effectively zero learning rate. This is why eval plateaued
+at step 130 — not capacity, not reward ceiling, not APO resistance. The optimizer ran out of
+gradient magnitude.
+
+```python
+# In GRPOConfig:
+lr_scheduler_type = "constant_with_warmup"
+warmup_ratio      = 0.05   # 5% warmup (30 steps out of 600)
+```
+
+---
+
+### Change 4: Raise the Learning Rate
+
+**What:** `LEARNING_RATE = 2e-6 → 5e-6`
+
+**Why:** V4's `train/grad_norm = 0.065` was low. The model can tolerate stronger updates.
+At 0.5B with LoRA r=16, 5e-6 is within the safe range per Dr. GRPO's Appendix G. Combined
+with the constant schedule, this gives sustained gradient magnitude throughout the 600-step run.
+
+---
+
+### V4.1 Config Summary
+
+```python
+MAX_STEPS              = 600       # was 200
+LEARNING_RATE          = 5e-6      # was 2e-6
+lr_scheduler_type      = "constant_with_warmup"   # was cosine
+warmup_ratio           = 0.05      # was 0.1 (keep short since constant schedule)
+
+# Everything else UNCHANGED from V4:
+NUM_GENERATIONS        = 16
+MAX_COMPLETION_LENGTH  = 512
+TEMPERATURE            = 1.0
+BETA                   = 0.0
+SCALE_REWARDS          = False
+BATCH_SIZE             = 2
+GRAD_ACCUM             = 1
+LORA_R                 = 16
+LORA_ALPHA             = 32
+```
+
+**Also:** Add per-task reward breakdown to `EvalRewardCallback`. This is essential for V4.2
+decisions — you need to know which specific tasks are gaining and which are stuck.
+
+```python
+# In EvalRewardCallback._run_eval(), track per-task:
+task_rewards = {"extraction": [], "sql_qa": [], "insights": [], "push": []}
+for record in subset:
+    msgs     = record["prompt"]
+    text     = tokenizer.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
+    user_txt = " ".join(m["content"] for m in msgs if m["role"] == "user")
+    task     = _classify_task_type(user_txt)
+    # ... generate response ...
+    r = self.reward_fn([response], [text])[0]
+    task_rewards[task].append(r)
+
+per_task = {t: sum(v)/len(v) for t, v in task_rewards.items() if v}
+wandb.log({"eval/" + k: v for k, v in per_task.items()}, step=state.global_step)
+```
+
+---
+
+## What to Observe During V4.1
+
+### Must watch in real time
+
+| Metric | Expected | Stop if |
+|---|---|---|
+| `eval/mean_reward` trend | Improving past step 130, continuing to ~step 400 | Plateaus before step 200 (same as V4) |
+| `eval/extraction` | Jumps significantly from V4's ~0.17 baseline | Still below 0.20 after step 100 (parser fix didn't help) |
+| `train/completion_length` | 100–400 tokens | Hits 512 ceiling (need to raise MAX_COMPLETION_LENGTH) |
+| `train/frac_reward_zero_std` | < 0.2 | Sustained > 0.5 |
+| `train/grad_norm` | 0.05–0.5 | Spikes > 2.0 (LR too high) |
+| `train/kl` | Any value (KL=0 is void at β=0) | Ignore entirely |
+
+### Questions to answer from the run
+
+1. **Did the parser fix change extraction reward?**  
+   Compare eval/extraction at step 10 (V4.1) vs V4's calibration baseline (~0.17). If it
+   jumps to 0.35+, the parser was the bottleneck. If it stays at ~0.17, the model genuinely
+   can't produce valid JSON at 0.5B.
+
+2. **Did eval continue improving past step 130?**  
+   The key question. If yes, the V4 plateau was entirely the LR schedule and not a capacity
+   limit. If no (plateaus at same ~0.476), the 0.5B model has a real ceiling.
+
+3. **Which task has the lowest per-task reward at step 600?**  
+   This determines the V4.2 priority. Whichever task scores lowest is either (a) reward
+   function too coarse, (b) 0.5B capacity insufficient, or (c) underrepresented in training.
+
+4. **What is eval/mean_reward at step 600?**  
+   Target: ≥ 0.55. If reached, scale to 3.7B. If stuck at ~0.48, proceed to V4.2.
+
+---
+
+## V4.2 — Conditional on V4.1 Results
+
+V4.2 depends entirely on what V4.1 tells you. Three scenarios:
+
+---
+
+### Scenario A: V4.1 reaches ≥ 0.55 eval reward
+
+**Conclusion:** 0.5B-Instruct is maximized. The recipe is validated.  
+**V4.2 = Scale to 3.7B-Instruct.** Apply the V4.1 config with these adjustments:
+
+```python
+MODEL_ID               = "Polygl0t/Tucano2-qwen-3.7B-Instruct"
+NUM_GENERATIONS        = 8         # VRAM constraint at 3.7B (was 16)
+MAX_COMPLETION_LENGTH  = 1024      # 3.7B can afford richer output (was 512)
+LEARNING_RATE          = 2e-6      # Larger model = smaller LR (was 5e-6)
+MAX_STEPS              = 400       # Fewer steps needed, larger model learns faster
+MAX_SEQ_LENGTH         = 2048      # Unchanged
+```
+
+Same reward functions, same schedule, same everything else. The qualitative findings transfer.
+Numerical hyperparameters don't — use the values above.
+
+---
+
+### Scenario B: V4.1 plateaus at ~0.476 (same as V4), eval/extraction still ≈ 0.17
+
+**Conclusion:** Parser fix didn't help — the model can't produce valid JSON at 0.5B.  
+**V4.2 = Switch base model.**
+
+Use `Polygl0t/Tucano2-qwen-0.5B-Base` (no APO, no SFT) with a minimal SFT warm-up first:
+
+```python
+# Step 1: 1-epoch SFT on extraction pairs only (~590 pairs)
+# 30 minutes on L4, teaches JSON output format
+MODEL_ID = "Polygl0t/Tucano2-qwen-0.5B-Base"
+# SFT with loss on assistant turns only, 1 epoch
+
+# Step 2: Same V4.1 GRPO config on top of SFT checkpoint
+# If extraction still fails after SFT warm-up → 0.5B genuinely can't do structured output
+# → skip 0.5B entirely, go directly to 3.7B-Instruct
+```
+
+---
+
+### Scenario C: V4.1 eval improves but one task type lags far behind others
+
+**Conclusion:** Task imbalance or reward function ceiling on the lagging task.  
+**V4.2 = Targeted fix for the weakest task.** Identify it from `eval/extraction`,
+`eval/sql_qa`, `eval/insights`, `eval/push` breakdown, then apply ONE of:
+
+**If extraction lags (reward < 0.25 at step 600):**  
+Reward function ceiling. Add semantic field-value scoring:
+```python
+# In reward_extraction: add bonus for correct field VALUES, not just field PRESENCE
+if data.get("sentiment") in VALID_SENTIMENTS:    score += 0.05  # was already there
+# Add: exact-match bonus against reference answer if available
+# Add: partial credit for correct complaint_category even if sentiment wrong
+```
+
+**If sql_qa lags (reward < 0.20 at step 600):**  
+Capacity limit at 0.5B for analytical tasks. Accept this and plan 3.7B for sql_qa.
+Keep 0.5B for extraction+push only, route sql_qa+insights to 3.7B in production.
+
+**If insights lags (reward < 0.20 at step 600):**  
+Most likely data starvation (only ~148 insights prompts in training set). Generate 200 more
+insights pairs from `commerce.db` before V4.2:
+```bash
+# Quick synthetic expansion using existing generate_pairs.py logic
+# Target: 350 total insights pairs (was 148)
+# Source: random sample from orders_enriched WHERE sentiment='negative'
+```
+
+---
+
+## Decision Tree
+
+```
+Run V4.1 (600 steps, 5e-6 LR, constant schedule, parser fix)
+        │
+        ├─ eval ≥ 0.55 at step 600?
+        │       YES → Scenario A: Scale to 3.7B-Instruct with V4.1 config
+        │
+        ├─ eval plateaus at ~0.476 AND extraction still ≈ 0.17?
+        │       YES → Scenario B: Switch to 0.5B-Base with SFT warm-up
+        │
+        └─ eval improves but one task type consistently < 0.20?
+                YES → Scenario C: Targeted fix for weakest task
+                      sql_qa weak → accept, plan 3.7B
+                      insights weak → generate more pairs
+                      extraction weak → refine reward function
+```
+
+---
+
+*V4 validated the recipe. V4.1 exhausts 0.5B before spending 3.7B compute.*