Create v4_2-handoff.md

(docs): V4.2 is the last 0.5B run. Its purpose is not to find more improvement — it is to know exactly what was found and why, with enough statistical rigor to say so in writing.

docs/v4_2-handoff.md

@@ -0,0 +1,450 @@
+# V4.2 Handoff — Closing the 0.5B to Gold Standard
+
+**Date:** 2026-04-27  
+**Context:** V4.1 achieved eval_best=0.645 (+35.5% over V4). Parser fix + constant LR
+were the two decisive changes. SQL Q&A remains stagnant (+3.8%). Insights swing
+(0.84→0.62) is eval noise, not regression. Goal: close remaining gaps before declaring
+the 0.5B scientifically complete and making the methodology case.
+
+---
+
+## What V4.1 Left Open (Ordered by Urgency)
+
+| Gap | Evidence | Blocks |
+|-----|----------|--------|
+| Eval suite too small (n≈2 for insights/push) | Insights swings ±0.22 between consecutive evals | Every subsequent claim about per-task performance |
+| SQL Q&A stagnant at +3.8% | Reward function doesn't validate SQL quality | Knowing whether this is reward ceiling or capacity limit |
+| Only 40% of training data seen | train/epoch=0.404 at step 600 | Knowing true performance ceiling |
+| No reward function audit protocol | Parser bug persisted 3 versions | Next parser-class bug catching you again |
+| Single run, no confidence intervals | No error bars on any reported number | Credibility of the methodology case |
+
+---
+
+## V4.2 Changes — Exactly What to Implement
+
+### Change 1: Expand Eval Suite to 50+ Samples (Do Before Training)
+
+**What:** Build a static eval set with minimum 15 samples per task type, stratified and
+held fixed across all future runs.
+
+```python
+EVAL_SAMPLES_PER_TASK = {
+    "extraction": 20,
+    "sql_qa":     15,
+    "insights":   15,
+    "push":       15,
+}
+# Total: 65 eval samples (was 15 mixed)
+```
+
+**How:** Sample from `data/pairs/eval.jsonl`, verify task distribution, save as
+`data/pairs/eval_v2_stratified.jsonl`. Never resample — the same 65 prompts must be
+used across V4.2 seeds 42, 123, 456.
+
+**Why this is Change 1:** The insights regression (0.84→0.62) made it impossible to
+distinguish learning from noise. At n≈2, standard error = ±0.22. At n=15, standard
+error drops to ±0.06. Every other change is uninterpretable without this fixed.
+
+**Report format:** `mean ± 1.96 × std/sqrt(n)` for 95% CI on each task.
+
+---
+
+### Change 2: Reward Function Audit Before Training (30-minute Protocol)
+
+**What:** Before launching any training cell, generate 20 completions (5 per task),
+manually score them 0-10, compute Spearman ρ between human scores and reward function.
+
+```python
+# Cell X — Reward Audit (add between calibration and training)
+from scipy.stats import spearmanr
+
+AUDIT_PROMPTS_PER_TASK = 5
+audit_human_scores = []   # filled manually by the person running the notebook
+audit_auto_scores  = []
+
+# Generate completions at temp=0.1 (deterministic), print each one
+# Person assigns 0-10 score, enters into audit_human_scores list
+# Then:
+rho, p_value = spearmanr(audit_human_scores, audit_auto_scores)
+print(f"Reward function calibration: ρ={rho:.2f} (p={p_value:.3f})")
+assert rho > 0.7, f"Reward function miscalibrated (ρ={rho:.2f} < 0.70). Fix before training."
+```
+
+**Gate:** ρ > 0.70. If below, find the discrepancy (another parser bug, wrong field
+weighting, wrong task classifier) before spending GPU hours.
+
+**Why:** The V1-V4 parser bug would have been caught in 30 minutes with this protocol.
+This is the single cheapest addition to the methodology.
+
+---
+
+### Change 3: SQL Reward Overhaul
+
+**What:** Replace heuristic vocabulary matching with a validation-aware reward that
+distinguishes "mentions SQL keywords" from "produces a correct analytical answer."
+
+```python
+def reward_sql_qa_v2(completion: str) -> float:
+    answer = strip_think(completion)
+    if not answer.strip():
+        return 0.0
+
+    score = 0.0
+
+    # Tier 1 (0.30): SQL structure detected
+    sql_keywords = ["SELECT", "FROM", "WHERE", "GROUP BY", "ORDER BY", "JOIN", "HAVING", "COUNT", "AVG", "SUM"]
+    sql_found = sum(1 for kw in sql_keywords if kw in answer.upper())
+    if sql_found >= 3:
+        score += 0.30
+    elif sql_found >= 1:
+        score += 0.15
+
+    # Tier 2 (0.25): Answer has both query and explanation
+    has_query  = bool(re.search(r"```sql|SELECT.{5,}FROM", answer, re.IGNORECASE | re.DOTALL))
+    has_answer = any(w in answer.lower() for w in ["resultado", "total", "média", "mostra", "portanto"])
+    if has_query and has_answer:
+        score += 0.25
+    elif has_query or has_answer:
+        score += 0.12
+
+    # Tier 3 (0.25): Numerical specificity
+    numbers = re.findall(r"\d+(?:[.,]\d+)?(?:\s*%)?", answer)
+    score += min(0.25, 0.05 * len(numbers))
+
+    # Tier 4 (0.20): Portuguese business domain coherence
+    pt_domain = ["pedidos", "clientes", "vendedores", "produtos", "avaliação",
+                 "entrega", "reclamação", "satisfação", "categoria", "período"]
+    score += min(0.20, 0.04 * sum(1 for w in pt_domain if w in answer.lower()))
+
+    return min(score, 1.0)
+```
+
+**Why:** The current SQL reward scores a completion that says "os pedidos de clientes em
+2017 totalizaram 15.000" the same as one that says "SELECT COUNT(*) FROM orders WHERE
+year=2017 → result: 15,000 pedidos". The V2 reward can't tell the difference. The model
+found the easier path (domain vocabulary + numbers) and stopped improving.
+
+**Expected outcome:** Two possibilities after the training run:
+- SQL reward improves +20%+ → reward was the ceiling, model has capacity
+- SQL reward still stagnates → confirmed 0.5B capacity limit for analytical reasoning
+
+Both are valid scientific conclusions. The point is to distinguish them.
+
+---
+
+### Change 4: Multi-Epoch Training (1,500 Steps)
+
+**What:** `MAX_STEPS = 600 → 1500`, training for ~2.5 full epochs with dataset shuffling.
+
+```python
+MAX_STEPS        = 1500    # was 600
+SAVE_STEPS       = 100     # was 50
+EVAL_STEPS       = 50      # was 20, now more frequent per-epoch boundary
+```
+
+**Why:** V4.1 saw only 40% of training data. The eval was still improving at step 500
+(best eval at step 500/600). There is signal remaining in the data. 1,500 steps covers
+the full dataset ~2× with shuffling between passes.
+
+**Risk monitoring:** Watch `eval/mean_reward` vs `train/reward` divergence. If eval
+plateaus while train keeps rising past step 800, the model is overfitting. Stop training
+and save the pre-overfitting checkpoint.
+
+**Shuffling:** Ensure `shuffle=True` in the DataLoader, or use a different seed per
+epoch for `Dataset.shuffle()`. Seeing the same prompts in the same order each epoch
+eliminates the diversity benefit.
+
+**Estimated cost:** ~12 hours on L4. Run overnight.
+
+---
+
+### Change 5: GDPO per-reward normalization
+
+```python
+# GDPO: normalize each reward component separately before summing
+# Instead of: reward = sum(extraction + sql + insights + push) → normalize batch
+# Do:         normalize each component independently, then sum
+
+def gdpo_normalize(component_rewards: dict[str, list[float]]) -> list[float]:
+    """Per-component normalization before aggregation (GDPO 2601.05242 §3.1)."""
+    normalized = {}
+    for task, rewards in component_rewards.items():
+        rewards_t = torch.tensor(rewards, dtype=torch.float32)
+        std = rewards_t.std()
+        if std > 1e-8:
+            normalized[task] = ((rewards_t - rewards_t.mean()) / std).tolist()
+        else:
+            normalized[task] = [0.0] * len(rewards)  # zero-variance group
+    # Sum normalized components per sample
+    n = len(next(iter(normalized.values())))
+    return [sum(normalized[t][i] for t in normalized) for i in range(n)]
+```
+
+This requires `commerce_reward_fn` to return per-component rewards (not just the sum), and
+the trainer to call `gdpo_normalize` before computing advantages. Moderate effort — requires
+a custom trainer subclass.
+
+---
+
+### Change 6: Dynamic Task Weighting (MT-GRPO IWU)
+
+**What:** Track per-task reward improvement rates every N eval steps and increase
+sampling probability for stagnating tasks.
+
+```python
+class DynamicTaskSampler:
+    """
+    MT-GRPO §3.2: Improvement-aware Weight Update.
+    Upweights tasks with stagnating reward, downweights converging tasks.
+    """
+    def __init__(self, tasks, initial_weight=0.25, update_interval=50):
+        self.tasks    = tasks
+        self.weights  = {t: initial_weight for t in tasks}
+        self.history  = {t: [] for t in tasks}
+        self.interval = update_interval
+
+    def update(self, step: int, per_task_rewards: dict):
+        if step % self.interval != 0 or step == 0:
+            return
+        for task, reward in per_task_rewards.items():
+            self.history[task].append(reward)
+            if len(self.history[task]) >= 2:
+                improvement = self.history[task][-1] - self.history[task][-2]
+                if improvement < 0.01:         # stagnating
+                    self.weights[task] = min(0.60, self.weights[task] * 1.3)
+                elif improvement > 0.05:       # improving fast
+                    self.weights[task] = max(0.10, self.weights[task] * 0.85)
+        # Normalize to sum to 1
+        total = sum(self.weights.values())
+        self.weights = {t: w / total for t, w in self.weights.items()}
+
+    def sample_indices(self, dataset, n_samples: int) -> list[int]:
+        """Sample indices with probability proportional to task weights."""
+        task_indices = {t: [] for t in self.tasks}
+        for i, record in enumerate(dataset):
+            user_txt = " ".join(m["content"] for m in record["prompt"] if m["role"] == "user")
+            task = _classify_task_type(user_txt)
+            task_indices[task].append(i)
+
+        sampled = []
+        for task, weight in self.weights.items():
+            n = max(1, int(n_samples * weight))
+            pool = task_indices.get(task, [])
+            if pool:
+                sampled.extend(random.sample(pool, min(n, len(pool))))
+        return sampled
+```
+
+**Why:** SQL Q&A improved +3.8% vs insights +68% under equal weighting. MT-GRPO's
+Theorem 1 proves GRPO allocates gradient budget to higher-variance tasks. SQL gets
+starved. IWU corrects this by progressively increasing SQL's sampling probability as
+its improvement rate drops below threshold.
+
+**Log:** `wandb.log({"sampler/sql_weight": sampler.weights["sql_qa"], ...})` at every
+update. You want to see SQL weight increasing from 0.40 (initial) toward 0.60+ if the
+model is still stagnating.
+
+---
+
+### Change 7: Three Seeds for Reproducibility
+
+**What:** Run V4.2 with `seed ∈ {42, 123, 456}`. Report `mean ± std` across seeds
+for all headline metrics.
+
+```python
+# In Cell 3:
+SEEDS = [42, 123, 456]
+CURRENT_SEED = 42   # change per run
+
+# In GRPOConfig and Dataset prep:
+seed = CURRENT_SEED
+random.seed(seed)
+torch.manual_seed(seed)
+```
+
+**Why:** Three seeds is the minimum for a credible ML result. A single lucky or unlucky
+random initialization can produce ±0.05 variance in eval reward. With three seeds, you
+can report `0.645 ± 0.02` instead of just `0.645` — the former is publishable, the
+latter is an observation.
+
+**Cost:** 3 × ~12h = ~36h on L4. Do these in parallel if you have access to multiple
+GPUs, or sequentially overnight.
+
+**Report format:**
+```
+| Task         | Seed 42 | Seed 123 | Seed 456 | Mean ± 95% CI |
+|---|---|---|---|---|
+| Extraction   | ...     | ...      | ...      | X.XX ± 0.0X   |
+| SQL Q&A      | ...     | ...      | ...      | X.XX ± 0.0X   |
+| Insights     | ...     | ...      | ...      | X.XX ± 0.0X   |
+| Push         | ...     | ...      | ...      | X.XX ± 0.0X   |
+| **Mean**     | ...     | ...      | ...      | **X.XX ± 0.0X** |
+```
+
+---
+
+### Change 8: Best checkpoint saving
+
+The V4.1 config had `save_total_limit=5` and `save_only_model=True` but no explicit best-checkpoint
+logic. `GRPOTrainer` doesn't have a native `load_best_model_at_end` equivalent the way `Trainer` does.
+The correct pattern is to track best eval reward in `EvalRewardCallback` and save explicitly when
+it improves:
+
+```python
+# In EvalRewardCallback.on_step_end, after computing mean_reward:
+if improved:
+    self.best_reward = mean_reward
+    self.best_step   = state.global_step
+    self.no_improve_count = 0
+    # Save best checkpoint explicitly
+    best_path = ADAPTER_DIR / "best_checkpoint"
+    model.save_pretrained(str(best_path))
+    tokenizer.save_pretrained(str(best_path))
+    print(f"  ✓ Best checkpoint saved → {best_path} (reward={mean_reward:.4f})")
+```
+
+This guarantees the adapter saved in `best_checkpoint/` is always the peak eval reward, regardless of where
+training ends or when early stopping fires. Add this to the `EvalRewardCallback` block and the final Cell 15
+should load from `best_checkpoint/` rather than the last training step.
+
+---
+
+## V4.2 Config Summary
+
+```python
+# Changes from V4.1
+MAX_STEPS                  = 1500     # was 600
+EVAL_STEPS                 = 50       # was 20
+SAVE_STEPS                 = 100      # was 50
+SEEDS                      = [42, 123, 456]  # run separately
+
+# Structural additions (not hyperparameters)
+eval_set                   = "data/pairs/eval_v2_stratified.jsonl"  # 65 fixed samples
+reward_sql_qa              = reward_sql_qa_v2  # new SQL reward (Change 3)
+dynamic_sampler            = DynamicTaskSampler(...)  # Change 5
+
+# Everything else UNCHANGED from V4.1 (these are validated):
+NUM_GENERATIONS            = 16
+MAX_COMPLETION_LENGTH      = 512
+TEMPERATURE                = 1.0
+BETA                       = 0.0
+SCALE_REWARDS              = False
+LEARNING_RATE              = 5e-6
+lr_scheduler_type          = "constant_with_warmup"
+warmup_ratio               = 0.05
+BATCH_SIZE                 = 2
+GRAD_ACCUM                 = 1
+LORA_R                     = 16
+LORA_ALPHA                 = 32
+```
+
+---
+
+## What to Observe During V4.2
+
+### The four questions V4.2 must answer
+
+**Q1: Does SQL reward improve with the new reward function?**  
+Watch `eval/sql_qa` trajectory. If it improves +15%+ over V4.1 by step 600, the old
+reward was the ceiling. If it still stagnates at ~0.547, the 0.5B model has a genuine
+capacity limit on analytical reasoning. Both are valid answers — the important thing is
+knowing which.
+
+**Q2: Is the insights regression noise or forgetting?**  
+With n=15 insights samples, the swing becomes ±0.06 instead of ±0.22. If insights
+stabilizes at 0.75-0.85 throughout multi-epoch training, it was noise. If it consistently
+drops after step 800, it's catastrophic forgetting — and you need MT-GRPO's constrained
+optimization to address it.
+
+**Q3: Does multi-epoch training push eval above 0.70?**  
+V4.1 was still improving at step 500 with eval_best=0.645. 1,500 steps of sustained
+training with constant LR and full data coverage should push this further. Target: ≥0.70
+mean across all tasks.
+
+**Q4: Are results reproducible across seeds?**  
+If `mean ± std` across three seeds shows std < 0.03, the result is robust. If std > 0.05,
+there's significant initialization sensitivity that needs to be understood before claiming
+these numbers.
+
+### WandB metrics to watch in real time
+
+| Metric | Expected | Stop if |
+|---|---|---|
+| `eval/sql_qa` | Steeper upward slope vs V4.1 | Still flat at step 200 (reward still wrong) |
+| `eval/insights` | Stable 0.75-0.85, no big swings | Drops consistently below 0.65 after step 800 |
+| `eval/mean_reward` | Continues improving past step 500 | Plateaus at same 0.645 as V4.1 |
+| `sampler/sql_weight` | Rising from 0.40 toward 0.55-0.60 | Stays flat (IWU not triggering) |
+| `train/loss` vs `eval` | Both decreasing | Train drops, eval plateaus (overfitting) |
+
+---
+
+## Drawing Conclusions for the Methodology Case
+
+V4.2 closes the 0.5B story. After the runs complete, the project demonstrates:
+
+### What was systematically learned across versions
+
+| Version | Failure identified | Fix | Isolated contribution |
+|---|---|---|---|
+| V1 | 68% extraction task imbalance → task collapse | Rebalanced to 40/40/10/10 | Established balanced multi-task training |
+| V2 | DPO near-no-op (9 gradient steps, flat loss) | Replaced DPO with GRPO | Established GRPO as the correct alignment method |
+| V3 | Think model APO anchor + 2,628-token `<think>` overhead | Switched to Instruct model | Isolated architectural constraint |
+| V4 | Cosine LR decay to zero by step 130 | Constant LR schedule | +370 productive training steps |
+| V4 | JSON parser failed on PT-BR decimals | json-repair + PT normalizer | 3.25× measured extraction improvement |
+| V4.1 | Stagnant SQL (+3.8%) | New reward function + dynamic weighting | To be measured in V4.2 |
+| V4.1 | Single run, no CIs | 3-seed reproducibility | Statistical credibility |
+
+### The scientific argument
+
+Each version was not "trying something different" — it was **testing a specific hypothesis**
+with a controlled change. The key methodological strength is causal attribution: when
+eval_best went from 0.476 to 0.645, the V4.1 report decomposed the sources:
+- ~0.13 from parser fix (measured at step 20, before GRPO learning)
+- ~0.13 from GRPO learning (measured from step 20 to peak)
+
+This is the scientific standard: not just "it improved" but "here is why, with evidence."
+
+V4.2 completes this arc by either showing SQL improvement (proving reward was the ceiling)
+or confirming capacity limit (proving 0.5B has an irreducible floor on analytical tasks).
+Either outcome is a clean, documentable finding.
+
+### The external benchmark gap
+
+One gap remains for true gold-standard credibility: no external benchmark comparison.
+The current eval suite is project-internal. For the methodology case, consider running
+the V4.2 final checkpoint against a subset of Portuguese NLP benchmarks from the Tucano2
+model card (BLUEX, OAB, ENEM) on the specific knowledge domains your training covered.
+Even partial coverage demonstrates the model didn't regress on general Portuguese ability
+while improving on domain tasks. This is the "catastrophic forgetting" test at the
+benchmark level, complementing the within-run insights stability test.
+
+---
+
+## Notebook Structure for V4.2
+
+```
+Cell  1: Dependencies + env vars
+Cell  2: GPU + Unsloth + TRL verification gate
+Cell  3: Config constants (CURRENT_SEED variable)
+Cell  4: Load model + critical generation_config overrides
+Cell  5: Token ID verification gate
+Cell  6: KV cache diagnostic gate
+Cell  7: Reward functions v2 (including reward_sql_qa_v2)
+Cell  8: Reward function audit (30-min protocol, ρ > 0.70 gate)
+Cell  9: Build stratified eval set (65 samples, fixed)
+Cell 10: Dataset preparation + DynamicTaskSampler init
+Cell 11: Smoke test (1 step, VRAM check)
+Cell 12: Probe run (10 steps, clip_ratio > 0 on 3/10 gate)
+Cell 13: W&B init + full training (1,500 steps)
+Cell 14: Post-training validation (stratified 65-sample eval)
+Cell 15: Save adapter
+Cell 16: Results table generation (for reporting)
+```
+
+**Run three times, changing only `CURRENT_SEED` in Cell 3.**
+
+---
+
+*V4.2 is the last 0.5B run. Its purpose is not to find more improvement —
+it is to know exactly what was found and why, with enough statistical rigor
+to say so in writing.*