ADR-002: V4 Instruct-Only GRPO — revises dual-model plan based on model repo audit

docs/ADR-002-v4-instruct.md

@@ -0,0 +1,1367 @@
+# ADR-002: V4 Instruct-Only GRPO — Single-Model Validation on 0.5B-Instruct
+
+**Status:** Proposed  
+**Date:** 2026-04-25  
+**Author:** Automated Investigation Agent  
+**Supersedes:** V4 Handoff (dual Instruct+Think plan)  
+**Context Documents:**
+- `docs/INVESTIGATION_REPORT.md` — full project audit (20+ papers)
+- `docs/ADR-001-next-steps.md` — V3 execution plans
+- `docs/checkpoints/2026-04-23_v3-launch.md` — V3 launch & probe results
+- V4 Handoff document (dual 0.5B hybrid plan)
+
+---
+
+## Table of Contents
+
+1. [Context: Why the Original V4 Plan Needs Revision](#1-context)
+2. [Decisions: What We Do Instead and Why](#2-decisions)
+3. [Consequences: What to Expect](#3-consequences)
+4. [Verified Model Facts: Tokenizer & Architecture Reference](#4-verified-model-facts)
+5. [Implementation: Cell-by-Cell Notebook Specification](#5-implementation)
+6. [Reward Functions: Complete Specification](#6-reward-functions)
+7. [Monitoring & Gate Conditions](#7-monitoring--gate-conditions)
+8. [Fallback Plan: What If Instruct Fails](#8-fallback-plan)
+9. [Hyperparameter Decision Log](#9-hyperparameter-decision-log)
+10. [File Structure](#10-file-structure)
+
+---
+
+## 1. Context
+
+### 1.1 V3 Autopsy (Confirmed)
+
+V3 ran 171 steps on `Polygl0t/Tucano2-qwen-3.7B-Think` and failed:
+
+```
+train/reward:              0.8675   ← high but policy didn't learn; SFT already does this
+train/clip_ratio:          0.0      ← zero on ALL 171 steps — policy never moved
+train/kl:                  0.159    ← tiny divergence from initialization
+train/completion_length:   2628     ← Think model fills context with <think>
+```
+
+The original V4 Handoff proposed a **dual-model** approach: train `Tucano2-qwen-0.5B-Instruct` for extraction+push, and `Tucano2-qwen-0.5B-Think` for SQL+insights, as two separate GRPO runs.
+
+### 1.2 Problems Found in the Original V4 Plan
+
+After auditing both model repos against their actual artifacts (`config.json`, `generation_config.json`, `chat_template.jinja`, `training_config_sft.yaml`, `training_config_apo.yaml`, `tokenizer_config.json`, `README.md` with benchmarks and inference samples), the following problems were identified:
+
+#### Problem 1: The 0.5B-Think Model Is Catastrophically Weak
+
+Published benchmarks from the model README:
+
+| Model | Total NPM | Knowledge & Reasoning NPM | GSM8K-PT | IFEval-PT |
+|---|---|---|---|---|
+| **0.5B-Instruct** | **26.08** | **27.77** | **18.49** | **30.00** |
+| **0.5B-Think** | **14.41** | **12.52** | **14.61** | **27.67** |
+
+The Think variant scores **nearly half** of Instruct on every benchmark. The Think SFT used only **34M tokens of reasoning data for 3,060 steps**, while Instruct SFT used **874M tokens across 9 task categories for 68,635 steps** — that is 25× less data and 22× fewer steps for Think.
+
+Concrete evidence from the Think model's own inference samples on the model card:
+- **Math (2x + 3 = 11):** The Think model's CoT arrives at the wrong answer (`x = 2` instead of `x = 4`). The model's thinking trace says "subtrairo 3 de ambos os lados" correctly, then in the final answer writes `x = 5 - 3 = 2` — it contradicts its own reasoning.
+- **Cooking recipe:** Hallucinates ingredient "30% ativo butylated buttercreme."
+- **History (Revolução Farroupilha):** Fabricates dates, events, and named entities that don't exist.
+
+By contrast, the Instruct model's inference samples show:
+- **Structured JSON output:** Correct, well-formatted extraction from an email.
+- **Math:** Correct solution (x = 4) without CoT.
+- **Function calling:** Correct tool-use JSON.
+- **Classification:** Correct sentiment classification.
+
+**Conclusion:** A model that cannot solve `2x + 3 = 11` is not a viable starting point for SQL analysis or business insights, even with GRPO tuning. GRPO refines what a model approximately knows — it cannot teach fundamentally new capabilities to a model this weak.
+
+#### Problem 2: Five Technical Bugs in the Original V4 Plan
+
+**Bug 2a: `use_cache: false` in both model configs.**
+
+Both `config.json` files ship with `"use_cache": false`. Without explicitly setting `model.config.use_cache = True` and `model.generation_config.use_cache = True` after loading, generation uses O(n²) full attention recomputation at every token. V3's notebook included this fix (Cell 4); the V4 plan omitted it entirely.
+
+**Source:** `Polygl0t/Tucano2-qwen-0.5B-Instruct/config.json`, line `"use_cache": false`.
+
+**Bug 2b: `repetition_penalty: 1.2` in `generation_config.json`.**
+
+Both models ship with `repetition_penalty: 1.2`. TRL's `GRPOTrainer` uses `model.generate()` internally for rollouts, and the `generation_config.json` defaults are loaded automatically. If `repetition_penalty` is not explicitly overridden to `1.0`, it will suppress diversity in rollout completions — directly working against GRPO's need for diverse outputs. The GRPOConfig `temperature` parameter overrides the generation config's temperature, but there is no `repetition_penalty` field in GRPOConfig. It must be overridden via `model.generation_config.repetition_penalty = 1.0` after model load.
+
+**Source:** `Polygl0t/Tucano2-qwen-0.5B-Instruct/generation_config.json`, line `"repetition_penalty": 1.2`.
+
+**Bug 2c: `temperature: 0.1` in `generation_config.json`.**
+
+Same as the V1 bug that destroyed the first training run. While GRPOConfig overrides temperature for rollouts, the model's generation_config may be used during eval callback generation if not explicitly overridden. Must set `model.generation_config.temperature = 1.0` as a defensive measure.
+
+**Source:** `Polygl0t/Tucano2-qwen-0.5B-Instruct/generation_config.json`, line `"temperature": 0.1`.
+
+**Bug 2d: Unsloth + tied word embeddings interaction.**
+
+Both 0.5B models have `"tie_word_embeddings": true` in config.json. When Unsloth applies LoRA, it targets linear projection layers. With tied embeddings, `embed_tokens` and `lm_head` share weights. If Unsloth's LoRA patching doesn't handle this correctly, gradients may not propagate to the output head, or the embedding table may drift independently. The smoke test (Cell 8) must verify that `model.lm_head.weight.data_ptr() == model.model.embed_tokens.weight.data_ptr()` holds after LoRA patching.
+
+**Source:** `Polygl0t/Tucano2-qwen-0.5B-Instruct/config.json`, line `"tie_word_embeddings": true`.
+
+**Bug 2e: The V4 plan's reference policy VRAM estimate may be wrong.**
+
+The V4 VRAM budget includes "Reference policy (frozen copy) ~0.4GB." With `beta=0.0` (no KL penalty), TRL 0.24.0's GRPOTrainer *may* skip loading the reference model entirely — the `ref_model` is only needed to compute KL divergence. But this behavior depends on the TRL version. If TRL loads the ref model anyway, it doubles the model footprint. This doesn't cause OOM at 0.5B (0.8GB total is fine), but it matters for the 3.7B scale-up. The smoke test must log peak VRAM to determine whether the ref model is loaded.
+
+#### Problem 3: Hyperparameter Transfer From 0.5B to 3.7B Is Overstated
+
+The original V4 plan claims hyperparameters validated at 0.5B transfer to 3.7B. This is partially true for qualitative findings (e.g., "clip_ratio > 0 is achievable," "task split works") but not for numerical values:
+- LR=2e-6 at 0.5B (490M params) likely needs LR=5e-7 at 3.7B (3.8B params) — smaller models tolerate higher LR.
+- G=16 at 0.5B is feasible with 512 completion length; at 3.7B the same VRAM budget supports G=4-8 at best.
+- Effective batch size effects differ: batch=32 at 0.5B vs batch=8-16 at 3.7B changes gradient noise characteristics.
+
+What transfers: the *qualitative* evidence that GRPO works on Tucano2-Instruct models, the reward function design, and the finding that APO-trained models can be further aligned (or can't).
+
+---
+
+## 2. Decisions
+
+### Decision 1: Single-Model, All-Task — 0.5B-Instruct Only
+
+**Decision:** Train one GRPO run on `Polygl0t/Tucano2-qwen-0.5B-Instruct` using ALL four task types (extraction, push, sql_qa, insights). Do not train the 0.5B-Think model.
+
+**Rationale:**
+1. The Instruct model scores 26.08 NPM vs Think's 14.41 — nearly 2× better on every benchmark.
+2. The Instruct model demonstrably produces correct structured JSON, correct math, correct function-call formatting (see model card samples).
+3. The Instruct chat template does NOT inject `<think>` tokens. The assistant message is just `{content}` — clean output, no token budget conflict.
+4. The Instruct model was SFT-trained on 874M tokens including structured output, retrieval, function calling, math with CoT, and general instruction following — it has a broad skill base suitable for all four tasks.
+5. Running one model simplifies the notebook, eliminates the data-split complexity, and halves the compute budget.
+6. If the Instruct model fails specifically on insights/analysis tasks, we can revisit Think for those tasks only. But the evidence says to test the strong model first.
+
+**Evidence:** ThinkJSON (2502.14905) demonstrated that a 1.5B Instruct/Base model + GRPO beats DeepSeek-R1-671B on JSON extraction. The Instruct model doesn't need CoT to do structured output well. For analytical tasks, GRPO's reward signal can teach the model to produce structured analysis without explicit `<think>` overhead.
+
+### Decision 2: Use ALL Training Data (No Task Split)
+
+**Decision:** Use the full V2 training set (`data/pairs/train.jsonl`, ~1,834 pairs) with the existing 40/40/10/10 distribution. Apply a 90/10 train/eval split. Do not create separate instruct/think data files.
+
+**Rationale:**
+1. More data = more diverse prompts = more GRPO signal. Splitting the data reduces each model's training set by ~50%.
+2. Multi-task training at this scale is a feature, not a bug — the Cocktail Effect paper (2410.01109) shows mixing task types improves domain performance by 2-15%.
+3. The reward function already dispatches by task type. GRPO handles mixed-task batches natively.
+
+### Decision 3: Override All Dangerous generation_config Defaults
+
+**Decision:** After model load, explicitly override the following `generation_config` fields:
+
+```python
+model.generation_config.temperature = 1.0
+model.generation_config.repetition_penalty = 1.0
+model.generation_config.do_sample = True
+model.generation_config.top_k = 0         # disable top-k during GRPO rollouts
+model.generation_config.top_p = 1.0       # disable top-p during GRPO rollouts
+model.config.use_cache = True
+model.generation_config.use_cache = True
+```
+
+**Rationale:**
+- `temperature=0.1` (default) destroyed V1. Must be overridden.
+- `repetition_penalty=1.2` (default) suppresses diversity. GRPO needs maximally diverse rollouts. Must be 1.0.
+- `top_k=50` and `top_p=1.0` are set in the default generation_config. `top_k=50` clips the distribution during sampling — at temp=1.0, this may unnecessarily restrict exploration. Set `top_k=0` (disabled) to let temperature alone control diversity.
+- `use_cache=false` (default) makes generation O(n²). Must be True.
+
+### Decision 4: Verify Tied Embeddings Survive LoRA Patching
+
+**Decision:** Add a verification cell after model loading that checks:
+
+```python
+# After Unsloth LoRA patching
+assert model.lm_head.weight.data_ptr() == model.model.embed_tokens.weight.data_ptr(), \
+    "CRITICAL: Tied embeddings broken after LoRA patching. lm_head and embed_tokens are now separate."
+```
+
+If this assertion fails, training may still work (gradients flow through LoRA layers on the projection matrices), but the embedding/output-head consistency that `tie_word_embeddings=true` provides would be broken. Document the result either way.
+
+### Decision 5: Hard Probe Gate on clip_ratio Before Full Training
+
+**Decision:** Run a 10-step probe. If `clip_ratio == 0.0` on all 10 steps, STOP. Do not proceed to full training. This was the missed signal in V3.
+
+**Gate condition:** `clip_ratio > 0.0` on **at least 3 of 10 probe steps**.
+
+If the gate fails, proceed to Fallback Plan (Section 8) — do not iterate blindly.
+
+### Decision 6: Strip `<think>` Defensively in All Reward Functions
+
+**Decision:** Even though the Instruct model's template doesn't inject `<think>`, the model may spontaneously generate think tokens (it has `<think>` as token ID 49116 in its vocabulary, and it was SFT-trained on math_cot data that contains reasoning traces). All reward functions must call `strip_think()` before scoring the answer portion.
+
+---
+
+## 3. Consequences
+
+### What We Expect
+
+| Metric | Expected Range | Justification |
+|---|---|---|
+| clip_ratio | > 0 on majority of steps | 0.5B model has fewer params → larger per-param gradient; G=16 → more reward variance; no `<think>` overhead → shorter completions → less gradient dilution |
+| Extraction reward | 0.30 - 0.60 | Instruct model already produces correct JSON (model card sample). GRPO refines schema compliance. |
+| Push reward | 0.40 - 0.70 | Short outputs, Portuguese heuristics — simple task at any scale. |
+| SQL Q&A reward | 0.20 - 0.40 | Model has general Portuguese comprehension. SQL-specific patterns need GRPO. Conservative target. |
+| Insights reward | 0.20 - 0.40 | Model can follow instructions and structure output. Domain-specific vocabulary needs GRPO. Conservative target. |
+| Completion length (Instruct) | 50 - 300 tokens | No `<think>` overhead. Extraction ~100 tok, SQL ~200 tok, insights ~300 tok. |
+| Training time | 3 - 6 hours | 0.5B is ~8× faster than 3.7B for generation. 200 steps × ~60-120s/step. |
+
+### What This Validates for 3.7B Scale-Up
+
+If V4 passes all gates:
+1. **GRPO works on APO-trained Tucano2 Instruct models.** The APO anchor resistance hypothesis is disproven.
+2. **All-task training on a single model is viable.** No need for a complex dual-model routing architecture.
+3. **Reward function calibration is confirmed.** The same reward functions (with appropriate thresholds) can be used at 3.7B.
+4. **The winning recipe:** 0.5B-Instruct + GRPO → scale to `Polygl0t/Tucano2-qwen-3.7B-Instruct` + GRPO.
+
+### What This Does NOT Validate
+
+- Exact hyperparameter values (LR, G, completion length) for 3.7B.
+- Whether 3.7B-Instruct has the same APO resistance characteristics as 0.5B-Instruct.
+- Whether 3.7B fits in L4 VRAM at the same G and completion length.
+
+---
+
+## 4. Verified Model Facts
+
+All values below were extracted directly from the actual repo files. The implementing agent should use these as ground truth, NOT the values from the original V4 handoff which contained some inaccuracies.
+
+### Tokenizer Token IDs (from `tokenizer_config.json`)
+
+```
+Token ID 0:     <|unk|>        (special=true)
+Token ID 1:     <|im_start|>   (special=true)  — bos_token
+Token ID 2:     <|im_end|>     (special=true)  — eos_token
+Token ID 49109: <|pad|>        (special=true)  — pad_token
+Token ID 49116: <think>        (special=false) — single token, NOT multi-token
+Token ID 49117: </think>       (special=false) — single token, NOT multi-token
+Token ID 49118: <answer>       (special=false)
+Token ID 49119: </answer>      (special=false)
+```
+
+**Critical note:** `<think>` (49116) and `</think>` (49117) are registered as **single dedicated tokens** in `added_tokens_decoder`. The original V4 plan warned that `</think>` might be multi-token — this is WRONG. It is a single token. Two-pass generation using `eos_token_id=[49117]` to stop at `</think>` IS technically feasible. However, we do not need two-pass generation because we are using the Instruct model which does not generate `<think>` by default.
+
+### Model Architecture (from `config.json`)
+
+```
+model_type:              qwen3
+architectures:           Qwen3ForCausalLM
+num_hidden_layers:       28
+hidden_size:             1024
+intermediate_size:       3072
+num_attention_heads:     16
+num_key_value_heads:     8
+head_dim:                128
+vocab_size:              49152
+max_position_embeddings: 4096
+tie_word_embeddings:     true
+use_cache:               false    ← MUST OVERRIDE TO true
+rope_theta:              1000000
+dtype:                   bfloat16
+Parameters:              490,799,104
+```
+
+### Generation Config (from `generation_config.json` — ALL must be overridden)
+
+```
+temperature:        0.1    ← OVERRIDE to 1.0 for training, 0.1 for eval
+repetition_penalty: 1.2    ← OVERRIDE to 1.0 for training
+do_sample:          true
+max_new_tokens:     1024
+eos_token_id:       [2]    (= <|im_end|>)
+```
+
+### Instruct Chat Template Behavior (from `chat_template.jinja`)
+
+The Instruct template applies the standard ChatML format:
+
+```
+<|im_start|>system
+{system_content}<|im_end|>
+<|im_start|>user
+{user_content}<|im_end|>
+<|im_start|>assistant
+{assistant_content}<|im_end|>
+```
+
+With `add_generation_prompt=True`, the template appends `<|im_start|>assistant\n` to prompt the model to generate. **There is no `<think>` injection anywhere** in the Instruct template. The assistant block is rendered as plain `{content}` without any reasoning wrapper.
+
+### APO Training Details (from `training_config_apo.yaml`)
+
+```
+loss_type:            apo_zero
+dpo_beta:             0.5
+max_steps:            1115
+max_learning_rate:    0.000005
+num_train_epochs:     5
+total_batch_size:     524288
+reference_model:      Tucano2-qwen-0.5B-Instruct-SFT
+precompute_ref_log_probs: true
+```
+
+The Instruct model had 1,115 steps of APO with `dpo_beta=0.5`. This is a moderate preference optimization — it creates a soft bias toward SFT behavior, not a hard constraint. With `beta=0.0` in GRPO (no KL penalty) and `LR=2e-6`, the GRPO gradient should be strong enough to move the policy.
+
+### SFT Training Details (from `training_config_sft.yaml`)
+
+```
+Data: 874M tokens across 9 categories:
+  - code:            ~2.3M tokens
+  - function_call:   ~17.5M tokens
+  - general:         ~700M tokens
+  - math_cot:        ~27M tokens
+  - retrieval:       ~2.2M tokens
+  - structured:      ~35M tokens
+  - summarization:   ~290K tokens
+  - translation:     ~5.7M tokens
+  - dpo (chosen):    ~14M tokens
+
+max_steps:           68,635
+max_learning_rate:   0.000085
+assistant_only_loss: true
+```
+
+The model was trained on structured output (35M tokens) and function calling (17.5M tokens) — it has a strong foundation for extraction tasks.
+
+---
+
+## 5. Implementation: Cell-by-Cell Notebook Specification
+
+The notebook is `v4_instruct_grpo.ipynb`. Each cell is a gate — verify output before proceeding.
+
+### Cell 1: Dependencies
+
+```python
+# Cell 1 — Clean install
+# Run after kernel restart
+
+!pip install "unsloth"
+!pip install "trl==0.24.0" --no-deps
+!pip install "rich" "wandb"
+```
+
+**Gate:** No errors. Verify TRL 0.24.0 installed.
+
+### Cell 2: GPU + Unsloth Verification
+
+```python
+import torch
+
+print(f"CUDA available: {torch.cuda.is_available()}")
+print(f"GPU: {torch.cuda.get_device_name(0)}")
+print(f"VRAM: {torch.cuda.get_device_properties(0).total_mem / 1e9:.1f} GB")
+print(f"bf16 support: {torch.cuda.is_bf16_supported()}")
+
+from unsloth import FastLanguageModel
+print(f"\n✓ Unsloth loaded")
+
+import trl
+assert trl.__version__ == "0.24.0", f"Expected TRL 0.24.0, got {trl.__version__}"
+print(f"✓ TRL {trl.__version__}")
+
+import transformers
+print(f"✓ Transformers {transformers.__version__}")
+```
+
+**Gate:** CUDA available, bf16=True, VRAM > 20GB, TRL 0.24.0.
+
+### Cell 3: Config Constants
+
+```python
+import os
+import json
+import re
+import time
+import random
+from pathlib import Path
+
+# ── Disable Unsloth kernel recompilation ─────────────────────────────────────
+os.environ["UNSLOTH_COMPILE_DISABLE"] = "1"
+os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True"
+
+# ── Model ────────────────────────────────────────────────────────────────────
+MODEL_ID       = "Polygl0t/Tucano2-qwen-0.5B-Instruct"
+MAX_SEQ_LENGTH = 2048   # model supports 4096, but 2048 is plenty for Instruct (no <think> overhead)
+ADAPTER_DIR    = Path("models/tucano2-0.5B-instruct-grpo-v4")
+CHECKPOINT_DIR = ADAPTER_DIR / "checkpoints"
+
+# ── Data ────────────��────────────────────────────────────────────────────────
+DATA_DIR       = Path("data/pairs")
+TRAIN_FILE     = DATA_DIR / "train.jsonl"
+EVAL_SPLIT     = 0.10   # 10% held out for eval
+
+# ── GRPO Hyperparameters ─────────────────────────────────────────────────────
+NUM_GENERATIONS        = 16     # 0.5B + short completions = VRAM allows G=16
+MAX_COMPLETION_LENGTH  = 512    # Instruct: no <think> overhead. Extraction ~100, SQL ~200, insights ~300
+TEMPERATURE            = 1.0    # Skywork-OR1: τ=1.0 for exploration
+LEARNING_RATE          = 2e-6   # Dr. GRPO: 4× V2's 5e-7 (clip_ratio=0 → push harder)
+BETA                   = 0.0    # Dr. GRPO §3.2: β=0 optimal for rule-based rewards
+SCALE_REWARDS          = False  # Dr. GRPO: remove std normalization bias
+BATCH_SIZE             = 2      # per-device batch size
+GRAD_ACCUM             = 1      # effective batch = 2 * 1 = 2 prompts * 16 gen = 32 completions
+MAX_STEPS              = 200    # validation run
+SAVE_STEPS             = 20
+EVAL_STEPS             = 10
+EARLY_STOPPING_PATIENCE = 15
+EARLY_STOPPING_DELTA   = 0.005
+
+# ── LoRA ─────────────────────────────────────────────────────────────────────
+LORA_R     = 16
+LORA_ALPHA = 32
+
+# ── Monitoring ───────────────────────────────────────────────────────────────
+WANDB_PROJECT     = "tucano2-commerce"
+EVAL_MAX_SAMPLES  = 15    # eval callback samples
+EVAL_MAX_TOKENS   = 512   # match training completion length
+
+# ── Task Classification (inherited from V2/V3) ──────────────────────────────
+VALID_SENTIMENTS  = {"positive", "negative", "neutral"}
+VALID_CATEGORIES  = {
+    "delivery_delay", "product_quality", "product_not_received",
+    "wrong_product", "seller_communication", "app_issue",
+    "price_value", "other", "none",
+}
+VALID_CHURN  = {"low", "medium", "high"}
+VALID_REPEAT = {"yes", "no", "maybe"}
+EXTRACTION_FIELDS = [
+    "sentiment", "sentiment_score", "churn_risk", "delivery_issue",
+    "product_issue", "seller_issue", "main_complaint",
+    "complaint_category", "repeat_intent", "would_recommend",
+]
+
+# ── Verified Special Token IDs (from tokenizer_config.json) ─────────────────
+# These are constants — do NOT recompute via tokenizer.encode()
+TOKEN_ID_BOS       = 1      # <|im_start|>
+TOKEN_ID_EOS       = 2      # <|im_end|>
+TOKEN_ID_PAD       = 49109  # <|pad|>
+TOKEN_ID_THINK     = 49116  # <think>
+TOKEN_ID_THINK_END = 49117  # </think>
+
+print("✓ Config loaded")
+print(f"  Model: {MODEL_ID}")
+print(f"  G={NUM_GENERATIONS}, max_comp={MAX_COMPLETION_LENGTH}, temp={TEMPERATURE}")
+print(f"  LR={LEARNING_RATE}, β={BETA}, scale_rewards={SCALE_REWARDS}")
+print(f"  LoRA r={LORA_R}, α={LORA_ALPHA}")
+print(f"  Max steps: {MAX_STEPS}")
+```
+
+### Cell 4: Load Model + Apply Critical Overrides
+
+```python
+from unsloth import FastLanguageModel
+
+print("Loading model...")
+model, tokenizer = FastLanguageModel.from_pretrained(
+    model_name=MODEL_ID,
+    max_seq_length=MAX_SEQ_LENGTH,
+    load_in_4bit=True,
+    dtype=None,  # auto-detect
+)
+
+# ═══════════════════════════════════════════════════════════════════════════════
+# CRITICAL OVERRIDES — generation_config ships with values that destroy GRPO
+# Source: Polygl0t/Tucano2-qwen-0.5B-Instruct/generation_config.json
+#   temperature: 0.1       → override to 1.0
+#   repetition_penalty: 1.2 → override to 1.0
+#   use_cache: false        → override to true
+# ═══════════════════════════════════════════════════════════════════════════════
+
+model.config.use_cache = True
+model.generation_config.use_cache = True
+model.generation_config.temperature = TEMPERATURE
+model.generation_config.repetition_penalty = 1.0   # CRITICAL: 1.2 suppresses diversity
+model.generation_config.do_sample = True
+model.generation_config.top_k = 0    # disable top-k — let temperature control diversity
+model.generation_config.top_p = 1.0  # disable top-p
+
+# Pad token
+if tokenizer.pad_token is None:
+    tokenizer.pad_token = tokenizer.eos_token
+
+print(f"✓ Model loaded on {model.device}")
+print(f"  use_cache: {model.config.use_cache}")
+print(f"  temperature: {model.generation_config.temperature}")
+print(f"  repetition_penalty: {model.generation_config.repetition_penalty}")
+print(f"  top_k: {model.generation_config.top_k}")
+print(f"  Params: {sum(p.numel() for p in model.parameters()) / 1e6:.0f}M")
+
+# ═══════════════════════════════════════════════════════════════════════════════
+# TIED EMBEDDINGS CHECK
+# Source: config.json has "tie_word_embeddings": true
+# If Unsloth LoRA patching breaks this, log it (may not be fatal).
+# ═══════════════════════════════════════════════════════════════════════════════
+
+try:
+    lm_ptr = model.lm_head.weight.data_ptr()
+    embed_ptr = model.model.embed_tokens.weight.data_ptr()
+    tied = lm_ptr == embed_ptr
+    print(f"  Tied embeddings intact: {tied}")
+    if not tied:
+        print("  ⚠️ WARNING: Tied embeddings broken after Unsloth load. May affect output head gradients.")
+except AttributeError as e:
+    print(f"  ⚠️ Could not check tied embeddings: {e}")
+```
+
+**Gate:** Model loaded, `use_cache=True`, `repetition_penalty=1.0`, `temperature=1.0`.
+
+### Cell 5: Token ID Verification
+
+```python
+# Verify that the constants from Cell 3 match the actual tokenizer
+# Do NOT skip this cell — if IDs don't match, all reward functions break
+
+tok_tests = {
+    "<|im_start|>": TOKEN_ID_BOS,
+    "<|im_end|>":   TOKEN_ID_EOS,
+    "<|pad|>":      TOKEN_ID_PAD,
+    "<think>":      TOKEN_ID_THINK,
+    "</think>":     TOKEN_ID_THINK_END,
+}
+
+all_pass = True
+for text, expected_id in tok_tests.items():
+    # For special tokens registered in added_tokens, encode should return single ID
+    ids = tokenizer.encode(text, add_special_tokens=False)
+    actual_id = ids[0] if len(ids) == 1 else ids
+    match = (len(ids) == 1 and ids[0] == expected_id)
+    status = "✓" if match else "✗"
+    print(f"  {status} '{text}' → expected {expected_id}, got {actual_id}")
+    if not match:
+        all_pass = False
+
+assert all_pass, "Token ID mismatch detected. Update constants in Cell 3 before proceeding."
+print("\n✓ All token IDs verified")
+
+# Also verify eos_token_id is correct
+assert tokenizer.eos_token_id == TOKEN_ID_EOS, f"eos_token_id mismatch: {tokenizer.eos_token_id}"
+print(f"✓ eos_token_id = {tokenizer.eos_token_id}")
+```
+
+**Gate:** All token IDs match. Single-token `<think>` (49116) and `</think>` (49117) confirmed.
+
+### Cell 6: KV Cache Diagnostic
+
+```python
+# Copied from V2 Cell 5b — verify KV cache is working
+# Gate: ratio < 3× → KV cache OK. ratio > 5× → BROKEN, abort.
+
+FastLanguageModel.for_inference(model)
+
+_kv_msgs = [{"role": "user", "content": "Qual a categoria de reclamação mais frequente?"}]
+_kv_text = tokenizer.apply_chat_template(_kv_msgs, tokenize=False, add_generation_prompt=True)
+_kv_inputs = tokenizer(_kv_text, return_tensors="pt").to(model.device)
+
+_token_times, _past, _generated = [], None, _kv_inputs["input_ids"]
+with torch.no_grad():
+    for _step in range(50):
+        _t0 = time.time()
+        seq_len = _generated.shape[1]
+        if _past is None:
+            _position_ids = torch.arange(seq_len, dtype=torch.long, device=model.device).unsqueeze(0)
+        else:
+            _position_ids = torch.tensor([[seq_len - 1]], dtype=torch.long, device=model.device)
+        _out = model(
+            input_ids=_generated[:, -1:] if _past else _generated,
+            position_ids=_position_ids,
+            attention_mask=torch.ones(1, seq_len, device=model.device),
+            past_key_values=_past,
+            use_cache=True,
+            return_dict=True,
+        )
+        _past = _out.past_key_values
+        _next = _out.logits[:, -1, :].argmax(dim=-1, keepdim=True)
+        _generated = torch.cat([_generated, _next], dim=1)
+        _token_times.append(time.time() - _t0)
+
+_ratio = sum(_token_times[45:]) / max(sum(_token_times[:5]), 1e-9)
+print(f"First 5 tok: {[f'{t*1000:.0f}ms' for t in _token_times[:5]]}")
+print(f"Last  5 tok: {[f'{t*1000:.0f}ms' for t in _token_times[45:]]}")
+print(f"Ratio last/first: {_ratio:.1f}x")
+assert _ratio < 5, f"KV cache BROKEN (ratio {_ratio:.1f}×). Check model.config.use_cache."
+print("✓ KV cache working correctly")
+
+del _past, _generated, _kv_inputs, _token_times, _out
+import gc; gc.collect()
+torch.cuda.empty_cache()
+```
+
+**Gate:** Ratio < 3×.
+
+### Cell 7: Single Inference Test
+
+```python
+# Verify model generates coherent Portuguese and closes <|im_end|>
+
+FastLanguageModel.for_inference(model)
+
+test_msgs = [
+    {"role": "system", "content": "Você é um assistente de IA especializado em e-commerce brasileiro."},
+    {"role": "user", "content": "Analise esta avaliação: 'Produto chegou quebrado, péssima embalagem. Nunca mais compro aqui.' Retorne um objeto JSON com os campos: sentiment, sentiment_score, delivery_issue, complaint_category."},
+]
+text = tokenizer.apply_chat_template(test_msgs, tokenize=False, add_generation_prompt=True)
+inputs = tokenizer(text, return_tensors="pt").to(model.device)
+
+t0 = time.time()
+outputs = model.generate(
+    **inputs,
+    max_new_tokens=256,
+    temperature=0.1,       # low temp for deterministic eval
+    do_sample=True,
+    repetition_penalty=1.0,
+)
+elapsed = time.time() - t0
+
+response = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
+print(f"Generation time: {elapsed:.1f}s")
+print(f"Response length: {len(response)} chars")
+print(f"Contains <think>: {'<think>' in response}")
+print(f"Contains JSON {{ }}: {'{' in response and '}' in response}")
+print(f"\n{'='*60}")
+print(response[:500])
+```
+
+**Gate:** Response is coherent Portuguese. Check whether `<think>` appears (document the result — this tells us if the Instruct model spontaneously thinks). Check if JSON structure is present.
+
+### Cell 8: Reward Functions
+
+Complete reward functions — see Section 6 below for the full specification. This cell defines:
+
+- `strip_think(text)` — remove `<think>...</think>` blocks
+- `has_think_block(text)` — check for think blocks
+- `_classify_task_type(prompt_text)` — classify prompt into task type
+- `_extract_json(text)` — extract JSON from text robustly
+- `reward_extraction(completion)` — continuous reward for JSON extraction (max 1.0)
+- `reward_sql_qa(completion)` — continuous reward for SQL Q&A (max 1.0)
+- `reward_insights(completion)` — continuous reward for insights (max 1.0)
+- `reward_push(completion)` — continuous reward for push notifications (max 1.0)
+- `commerce_reward_fn(completions, prompts, **kwargs)` — master dispatch function
+
+### Cell 9: Reward Calibration
+
+```python
+# Load data, classify by task type, run calibration on 8 diverse samples
+
+by_type = {"extraction": [], "sql_qa": [], "insights": [], "push": []}
+with open(TRAIN_FILE) as f:
+    for line in f:
+        row = json.loads(line)
+        convs = row["conversations"]
+        prompt_msgs = [m for m in convs if m["role"] in ("system", "user")]
+        if not prompt_msgs:
+            continue
+        user_text = " ".join(m["content"] for m in prompt_msgs if m["role"] == "user")
+        task = _classify_task_type(user_text)
+        by_type[task].append(prompt_msgs)
+
+print(f"Prompts by type: {', '.join(f'{k}={len(v)}' for k, v in by_type.items())}")
+
+# Pick 2 samples per task type = 8 total
+rng = random.Random(42)
+cal_samples = []
+for task_type in by_type:
+    pool = by_type[task_type]
+    if len(pool) >= 2:
+        cal_samples.extend(rng.sample(pool, 2))
+    elif pool:
+        cal_samples.extend(pool)
+
+FastLanguageModel.for_inference(model)
+print(f"\nReward calibration ({len(cal_samples)} samples):")
+print("-" * 60)
+
+cal_rewards = []
+for i, msgs in enumerate(cal_samples):
+    text = tokenizer.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
+    inputs = tokenizer(text, return_tensors="pt").to(model.device)
+    outputs = model.generate(
+        **inputs,
+        max_new_tokens=MAX_COMPLETION_LENGTH,
+        temperature=0.7,
+        do_sample=True,
+        repetition_penalty=1.0,
+    )
+    response = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
+    r = commerce_reward_fn([response], [text])[0]
+    cal_rewards.append(r)
+    task = _classify_task_type(" ".join(m.get("content", "") for m in msgs if m["role"] == "user"))
+    has_think = "<think>" in response
+    answer_preview = strip_think(response)[:100]
+    print(f"  Sample {i+1} [{task:12s}]: reward={r:.2f} | has_think={has_think} | {answer_preview}")
+
+print(f"\nMean={sum(cal_rewards)/len(cal_rewards):.2f}, Min={min(cal_rewards):.2f}, Max={max(cal_rewards):.2f}")
+print(f"Reward variance > 0: {len(set(f'{r:.4f}' for r in cal_rewards)) > 1}")
+```
+
+**Gate:** Mean reward < 0.90 (if already ~1.0, the reward function is too easy — GRPO won't learn). Variance > 0. Document whether `<think>` appeared.
+
+### Cell 10: Dataset Preparation
+
+```python
+from datasets import Dataset
+
+def prepare_datasets(train_file, eval_ratio=EVAL_SPLIT, seed=42):
+    rng = random.Random(seed)
+    
+    all_records = []
+    with open(train_file) as f:
+        for line in f:
+            row = json.loads(line)
+            convs = row["conversations"]
+            prompt_msgs = [m for m in convs if m["role"] in ("system", "user")]
+            if prompt_msgs:
+                all_records.append(prompt_msgs)
+    
+    rng.shuffle(all_records)
+    n_eval = max(1, int(len(all_records) * eval_ratio))
+    eval_records = all_records[:n_eval]
+    train_records = all_records[n_eval:]
+    
+    # Log task distribution
+    for label, records in [("train", train_records), ("eval", eval_records)]:
+        dist = {}
+        for msgs in records:
+            user_text = " ".join(m["content"] for m in msgs if m["role"] == "user")
+            task = _classify_task_type(user_text)
+            dist[task] = dist.get(task, 0) + 1
+        print(f"  {label}: {len(records)} prompts — {dist}")
+    
+    train_ds = Dataset.from_list([{"prompt": msgs} for msgs in train_records])
+    eval_ds = Dataset.from_list([{"prompt": msgs} for msgs in eval_records])
+    return train_ds, eval_ds
+
+train_dataset, eval_dataset = prepare_datasets(TRAIN_FILE)
+print(f"\n✓ Datasets: train={len(train_dataset)}, eval={len(eval_dataset)}")
+```
+
+**Gate:** Train has ~1,650 prompts, eval has ~180. All 4 task types present in both.
+
+### Cell 11: Smoke Test (1 Step)
+
+```python
+from trl import GRPOConfig, GRPOTrainer
+
+FastLanguageModel.for_training(model)
+
+smoke_config = GRPOConfig(
+    output_dir=str(CHECKPOINT_DIR / "smoke"),
+    num_generations=NUM_GENERATIONS,
+    scale_rewards=SCALE_REWARDS,
+    max_completion_length=MAX_COMPLETION_LENGTH,
+    max_steps=1,
+    temperature=TEMPERATURE,
+    per_device_train_batch_size=BATCH_SIZE,
+    gradient_accumulation_steps=1,
+    learning_rate=LEARNING_RATE,
+    fp16=False,
+    bf16=True,
+    logging_steps=1,
+    save_steps=999,
+    report_to="none",
+    max_prompt_length=MAX_SEQ_LENGTH // 2,
+    seed=42,
+    remove_unused_columns=False,
+)
+
+# ── UnslothGRPOTrainer (inherited from V2/V3) ────────────────────────────────
+class UnslothGRPOTrainer(GRPOTrainer):
+    def _generate(self, prompts, images):
+        FastLanguageModel.for_inference(self.model)
+        try:
+            result = super()._generate(prompts, images)
+        finally:
+            FastLanguageModel.for_training(self.model)
+        return result
+
+smoke_trainer = UnslothGRPOTrainer(
+    model=model,
+    reward_funcs=commerce_reward_fn,
+    args=smoke_config,
+    train_dataset=train_dataset,
+    processing_class=tokenizer,
+)
+
+t0 = time.time()
+smoke_trainer.train()
+step_time = time.time() - t0
+
+peak_vram = torch.cuda.max_memory_allocated() / 1e9
+print(f"\n✓ Smoke test passed!")
+print(f"  Step time: {step_time:.0f}s")
+print(f"  Peak VRAM: {peak_vram:.1f}GB / {torch.cuda.get_device_properties(0).total_mem / 1e9:.1f}GB")
+print(f"  Estimated full run ({MAX_STEPS} steps): {step_time * MAX_STEPS / 3600:.1f}h")
+
+del smoke_trainer
+gc.collect(); torch.cuda.empty_cache()
+```
+
+**Gate:** No OOM. Peak VRAM < 20GB. Step time < 180s. Document whether ref model was loaded (check VRAM: if peak > 1.0GB, ref model is loaded; if ~0.5GB, it's skipped due to β=0).
+
+### Cell 12: Probe Run (10 Steps) — THE CRITICAL GATE
+
+```python
+FastLanguageModel.for_training(model)
+
+probe_config = GRPOConfig(
+    output_dir=str(CHECKPOINT_DIR / "probe"),
+    num_generations=NUM_GENERATIONS,
+    scale_rewards=SCALE_REWARDS,
+    max_completion_length=MAX_COMPLETION_LENGTH,
+    max_steps=10,
+    temperature=TEMPERATURE,
+    num_train_epochs=1,
+    per_device_train_batch_size=BATCH_SIZE,
+    gradient_accumulation_steps=GRAD_ACCUM,
+    learning_rate=LEARNING_RATE,
+    warmup_ratio=0.1,
+    lr_scheduler_type="cosine",
+    fp16=False,
+    bf16=True,
+    logging_steps=1,
+    save_steps=999,
+    report_to="none",
+    max_prompt_length=MAX_SEQ_LENGTH // 2,
+    seed=42,
+    remove_unused_columns=False,
+)
+
+probe_trainer = UnslothGRPOTrainer(
+    model=model,
+    reward_funcs=commerce_reward_fn,
+    args=probe_config,
+    train_dataset=train_dataset,
+    processing_class=tokenizer,
+)
+
+t0 = time.time()
+result = probe_trainer.train()
+elapsed = time.time() - t0
+
+# ══════════════════════════════════════════════════════════════════════════════
+# CRITICAL GATE: clip_ratio > 0 on at least 3 of 10 steps
+# If this fails, STOP. See Fallback Plan (Section 8 of ADR-002).
+# ══════════════════════════════════════════════════════════════════════════════
+# TRL logs clip_ratio in training history. Extract from trainer.state.log_history.
+clip_ratios = []
+for entry in probe_trainer.state.log_history:
+    if "train/clip_ratio" in entry:
+        clip_ratios.append(entry["train/clip_ratio"])
+
+nonzero_clips = sum(1 for cr in clip_ratios if cr > 0.0)
+print(f"\n{'='*60}")
+print(f"PROBE RESULTS ({elapsed:.0f}s, {elapsed/10:.0f}s/step)")
+print(f"  clip_ratios: {[f'{cr:.4f}' for cr in clip_ratios]}")
+print(f"  Non-zero clip steps: {nonzero_clips}/{len(clip_ratios)}")
+print(f"  Train loss: {result.training_loss:.4f}")
+print(f"{'='*60}")
+
+if nonzero_clips >= 3:
+    print("✓ PROBE GATE PASSED — proceed to full training")
+elif nonzero_clips > 0:
+    print("⚠️ MARGINAL — clip_ratio > 0 on some steps but < 3. Consider increasing LR or G.")
+else:
+    print("✗ PROBE GATE FAILED — clip_ratio = 0 on ALL steps.")
+    print("  DO NOT proceed to full training.")
+    print("  See ADR-002 Section 8 (Fallback Plan).")
+
+del probe_trainer
+gc.collect(); torch.cuda.empty_cache()
+```
+
+**Gate:** `nonzero_clips >= 3`. If this fails, go to Section 8.
+
+### Cell 13: W&B Init + Full Training
+
+```python
+import wandb
+
+wandb.login()
+wandb.init(
+    project=WANDB_PROJECT,
+    name=f"grpo-v4-instruct-0.5B-{time.strftime('%Y%m%d-%H%M')}",
+    config={
+        "model_id": MODEL_ID,
+        "version": "v4",
+        "num_generations": NUM_GENERATIONS,
+        "max_completion_length": MAX_COMPLETION_LENGTH,
+        "temperature": TEMPERATURE,
+        "learning_rate": LEARNING_RATE,
+        "beta": BETA,
+        "scale_rewards": SCALE_REWARDS,
+        "batch_size": BATCH_SIZE,
+        "grad_accum": GRAD_ACCUM,
+        "max_steps": MAX_STEPS,
+        "lora_r": LORA_R,
+        "lora_alpha": LORA_ALPHA,
+        "train_prompts": len(train_dataset),
+        "eval_prompts": len(eval_dataset),
+        "repetition_penalty_override": 1.0,
+    },
+)
+print(f"✓ W&B run: {wandb.run.url}")
+
+# ── EvalRewardCallback (inherited from V2/V3, adapted) ──────────────────────
+from transformers import TrainerCallback
+
+class EvalRewardCallback(TrainerCallback):
+    def __init__(self, eval_records, reward_fn, patience, delta):
+        self.eval_records = eval_records
+        self.reward_fn = reward_fn
+        self.patience = patience
+        self.delta = delta
+        self.best_reward = -float("inf")
+        self.best_step = 0
+        self.no_improve_count = 0
+
+    def on_step_end(self, args, state, control, model=None, processing_class=None, **kwargs):
+        if state.global_step == 0 or state.global_step % EVAL_STEPS != 0:
+            return control
+
+        tokenizer_local = processing_class
+        if tokenizer_local is None:
+            print("[EvalRewardCallback] WARNING: tokenizer is None, skipping eval")
+            return control
+
+        mean_reward = self._run_eval(model, tokenizer_local, args)
+        improved = mean_reward > self.best_reward + self.delta
+
+        wandb.log({
+            "eval/mean_reward": mean_reward,
+            "eval/best_reward": max(self.best_reward, mean_reward),
+            "eval/no_improve_count": self.no_improve_count,
+        }, step=state.global_step)
+
+        status = "↑ improved" if improved else f"↔ no gain ({self.no_improve_count + 1}/{self.patience})"
+        print(f"\n[EvalReward] step={state.global_step} | mean={mean_reward:.4f} | best={self.best_reward:.4f} | {status}")
+
+        if improved:
+            self.best_reward = mean_reward
+            self.best_step = state.global_step
+            self.no_improve_count = 0
+        else:
+            self.no_improve_count += 1
+            if self.no_improve_count >= self.patience:
+                print(f"[EarlyStopping] No improvement for {self.patience} evals. Halting.")
+                control.should_training_stop = True
+        return control
+
+    def _run_eval(self, model, tokenizer_local, args):
+        FastLanguageModel.for_inference(model)
+        rewards = []
+        subset = self.eval_records[:EVAL_MAX_SAMPLES]
+        for record in subset:
+            msgs = record["prompt"]
+            text = tokenizer_local.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
+            inputs = tokenizer_local(text, return_tensors="pt", truncation=True, max_length=args.max_prompt_length).to(model.device)
+            with torch.no_grad():
+                out = model.generate(
+                    **inputs,
+                    max_new_tokens=EVAL_MAX_TOKENS,
+                    temperature=0.1,   # deterministic eval
+                    do_sample=True,
+                    repetition_penalty=1.0,
+                )
+            resp = tokenizer_local.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
+            rewards.append(self.reward_fn([resp], [text])[0])
+        FastLanguageModel.for_training(model)
+        return sum(rewards) / len(rewards) if rewards else 0.0
+
+# ── Training ────────────────────────────────────────────────────────────────
+FastLanguageModel.for_training(model)
+
+grpo_config = GRPOConfig(
+    output_dir=str(CHECKPOINT_DIR),
+    num_generations=NUM_GENERATIONS,
+    scale_rewards=SCALE_REWARDS,
+    max_completion_length=MAX_COMPLETION_LENGTH,
+    max_steps=MAX_STEPS,
+    temperature=TEMPERATURE,
+    num_train_epochs=1,
+    per_device_train_batch_size=BATCH_SIZE,
+    gradient_accumulation_steps=GRAD_ACCUM,
+    learning_rate=LEARNING_RATE,
+    warmup_ratio=0.1,
+    lr_scheduler_type="cosine",
+    fp16=False,
+    bf16=True,
+    logging_steps=1,
+    save_steps=SAVE_STEPS,
+    save_total_limit=5,
+    save_only_model=True,
+    report_to="wandb",
+    max_prompt_length=MAX_SEQ_LENGTH // 2,
+    seed=42,
+    remove_unused_columns=False,
+    disable_tqdm=True,
+    logging_first_step=True,
+)
+
+eval_cb = EvalRewardCallback(
+    eval_records=list(eval_dataset),
+    reward_fn=commerce_reward_fn,
+    patience=EARLY_STOPPING_PATIENCE,
+    delta=EARLY_STOPPING_DELTA,
+)
+
+trainer = UnslothGRPOTrainer(
+    model=model,
+    reward_funcs=commerce_reward_fn,
+    args=grpo_config,
+    train_dataset=train_dataset,
+    processing_class=tokenizer,
+    callbacks=[eval_cb],
+)
+
+t_start = time.time()
+result = trainer.train()
+elapsed = time.time() - t_start
+
+wandb.log({
+    "train/final_loss": result.training_loss,
+    "train/duration_hours": elapsed / 3600,
+    "train/total_steps": result.global_step,
+    "eval/best_reward_final": eval_cb.best_reward,
+    "eval/best_step": eval_cb.best_step,
+})
+wandb.finish()
+
+print(f"\n{'='*60}")
+print(f"V4 Training Complete")
+print(f"  Loss:        {result.training_loss:.4f}")
+print(f"  Steps:       {result.global_step}")
+print(f"  Duration:    {elapsed/3600:.1f}h")
+print(f"  Best eval:   {eval_cb.best_reward:.4f} (step {eval_cb.best_step})")
+print(f"{'='*60}")
+```
+
+### Cell 14: Validation (20 Held-Out Samples)
+
+```python
+# Run validation on 20 held-out samples, broken down by task type
+
+FastLanguageModel.for_inference(model)
+
+val_samples = list(eval_dataset)[:20]
+val_results = {"extraction": [], "sql_qa": [], "insights": [], "push": []}
+
+for i, record in enumerate(val_samples):
+    msgs = record["prompt"]
+    user_text = " ".join(m["content"] for m in msgs if m["role"] == "user")
+    task = _classify_task_type(user_text)
+    
+    text = tokenizer.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
+    inputs = tokenizer(text, return_tensors="pt").to(model.device)
+    with torch.no_grad():
+        out = model.generate(
+            **inputs,
+            max_new_tokens=MAX_COMPLETION_LENGTH,
+            temperature=0.1,
+            do_sample=True,
+            repetition_penalty=1.0,
+        )
+    resp = tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
+    r = commerce_reward_fn([resp], [text])[0]
+    val_results[task].append(r)
+    print(f"  [{task:12s}] reward={r:.2f} | {strip_think(resp)[:80]}")
+
+print(f"\n{'='*60}")
+print("Validation Results by Task:")
+for task, rewards in val_results.items():
+    if rewards:
+        mean_r = sum(rewards) / len(rewards)
+        print(f"  {task:12s}: mean={mean_r:.3f} (n={len(rewards)})")
+print(f"{'='*60}")
+```
+
+### Cell 15: Save Adapter
+
+```python
+# Save the GRPO-tuned LoRA adapter
+
+model.save_pretrained(str(ADAPTER_DIR))
+tokenizer.save_pretrained(str(ADAPTER_DIR))
+print(f"✓ Adapter saved to {ADAPTER_DIR}")
+```
+
+---
+
+## 6. Reward Functions: Complete Specification
+
+These are the exact reward functions the implementing agent must use. They are adapted from V2/V3 with one critical change: `strip_think()` is called defensively on ALL completions before scoring, even for the Instruct model.
+
+```python
+def strip_think(text: str) -> str:
+    """Remove <think>...</think> block, return the answer portion."""
+    return re.sub(r"<think>.*?</think>", "", text, flags=re.DOTALL).strip()
+
+def has_think_block(text: str) -> bool:
+    return bool(re.search(r"<think>.+</think>", text, flags=re.DOTALL))
+
+def _classify_task_type(prompt_text: str) -> str:
+    p = prompt_text.lower()
+    if "retorne um objeto json" in p or "extraia dados" in p or "json" in p:
+        return "extraction"
+    elif "notificação push" in p or "notificação de reengajamento" in p:
+        return "push"
+    elif "perfil do cliente" in p or "retenção" in p or "análise" in p or "insight" in p:
+        return "insights"
+    else:
+        return "sql_qa"
+
+def _extract_json(text: str) -> dict | None:
+    """Extract first JSON object from text. Returns parsed dict or None."""
+    # Try direct parse first
+    stripped = text.strip()
+    # Remove markdown code blocks if present
+    stripped = re.sub(r"^```(?:json)?\s*", "", stripped)
+    stripped = re.sub(r"\s*```$", "", stripped)
+    stripped = stripped.strip()
+    try:
+        return json.loads(stripped)
+    except (json.JSONDecodeError, TypeError):
+        pass
+    # Try to find JSON object within text
+    match = re.search(r"\{[^{}]*(?:\{[^{}]*\}[^{}]*)*\}", text, re.DOTALL)
+    if match:
+        try:
+            return json.loads(match.group())
+        except (json.JSONDecodeError, TypeError):
+            pass
+    return None
+
+def reward_extraction(completion: str) -> float:
+    """Continuous reward for extraction tasks (max 1.0)."""
+    answer = strip_think(completion)
+    data = _extract_json(answer)
+    
+    if data is None:
+        # Partial credit for JSON-like structure
+        if "{" in answer and "}" in answer:
+            return 0.05
+        return 0.0
+    
+    if not isinstance(data, dict):
+        return 0.1  # valid JSON but not an object
+    
+    score = 0.3  # valid JSON object
+    
+    # Schema completeness (0.3 total)
+    present = sum(1 for f in EXTRACTION_FIELDS if f in data)
+    score += 0.3 * (present / len(EXTRACTION_FIELDS))
+    
+    # Value validity (0.4 total, split across checks)
+    checks_passed = 0
+    checks_total = 0
+    
+    for field, validator in [
+        ("sentiment", lambda v: v in VALID_SENTIMENTS),
+        ("complaint_category", lambda v: v in VALID_CATEGORIES),
+        ("churn_risk", lambda v: v in VALID_CHURN),
+        ("repeat_intent", lambda v: v in VALID_REPEAT),
+        ("sentiment_score", lambda v: isinstance(v, (int, float)) and 1 <= v <= 5),
+    ]:
+        checks_total += 1
+        if field in data and validator(data[field]):
+            checks_passed += 1
+    
+    for bool_field in ("delivery_issue", "product_issue", "seller_issue", "would_recommend"):
+        checks_total += 1
+        if bool_field in data and isinstance(data[bool_field], bool):
+            checks_passed += 1
+    
+    if checks_total > 0:
+        score += 0.4 * (checks_passed / checks_total)
+    
+    return min(score, 1.0)
+
+def reward_sql_qa(completion: str) -> float:
+    """Continuous reward for SQL Q&A (max 1.0)."""
+    answer = strip_think(completion)
+    if not answer.strip():
+        return 0.0
+    
+    score = 0.0
+    
+    # Numerical content (more numbers = more specific answer)
+    numbers = re.findall(r"\d+(?:[.,]\d+)?", answer)
+    score += min(0.4, 0.1 * len(numbers))
+    
+    # Length: 50-500 chars optimal
+    length = len(answer)
+    if 50 <= length <= 500:
+        score += 0.3
+    elif length > 0:
+        score += 0.3 * max(0, 1 - abs(length - 275) / 275)
+    
+    # Portuguese business vocabulary
+    pt_business = ["pedidos", "clientes", "média", "total", "taxa", "vendas",
+                   "produtos", "período", "categoria", "região", "faturamento"]
+    pt_matches = sum(1 for w in pt_business if w in answer.lower())
+    score += min(0.3, 0.06 * pt_matches)
+    
+    return min(score, 1.0)
+
+def reward_insights(completion: str) -> float:
+    """Continuous reward for insights (max 1.0)."""
+    answer = strip_think(completion)
+    if not answer.strip():
+        return 0.0
+    
+    score = 0.0
+    
+    # Actionable language
+    action_words = ["recomend", "implement", "melhor", "reduzir", "aumentar",
+                    "priorizar", "investir", "otimizar", "estratégi", "ação"]
+    matches = sum(1 for w in action_words if w in answer.lower())
+    score += min(0.4, 0.08 * matches)
+    
+    # Length: 100-800 chars optimal
+    length = len(answer)
+    if 100 <= length <= 800:
+        score += 0.3
+    elif length > 0:
+        score += 0.3 * max(0, 1 - abs(length - 450) / 450)
+    
+    # Structure: bullet points, numbered lists, headers
+    structure_marks = len(re.findall(r"^[-•*]\s|^\d+[.)]\s|^#{1,3}\s", answer, re.MULTILINE))
+    score += min(0.2, 0.04 * structure_marks)
+    
+    # Portuguese coherence marker
+    if any(w in answer.lower() for w in ["cliente", "produto", "serviço", "empresa"]):
+        score += 0.1
+    
+    return min(score, 1.0)
+
+def reward_push(completion: str) -> float:
+    """Continuous reward for push notifications (max 1.0)."""
+    answer = strip_think(completion).strip()
+    if not answer:
+        return 0.0
+    
+    # Length: ≤120 chars gets full credit
+    length = len(answer)
+    if length <= 120:
+        length_score = 0.5
+    else:
+        length_score = 0.5 * max(0, 1 - (length - 120) / 120)
+    
+    # Portuguese content
+    pt_markers = re.findall(r"[ãçéêóúâõ]|você|para|como|seu|sua|oferta|desconto|produto",
+                            answer, re.IGNORECASE)
+    lang_score = min(0.3, 0.03 * len(pt_markers))
+    
+    # Non-generic (penalize very generic phrases)
+    generic = ["olá", "obrigado pela compra", "agradecemos"]
+    is_generic = any(g in answer.lower() for g in generic)
+    creativity_score = 0.0 if is_generic else 0.2
+    
+    return min(length_score + lang_score + creativity_score, 1.0)
+
+def commerce_reward_fn(completions, prompts, **kwargs) -> list[float]:
+    """Master reward function: dispatches by task type."""
+    rewards = []
+    for completion, prompt in zip(completions, prompts):
+        if isinstance(completion, list):
+            comp_text = completion[-1]["content"] if completion else ""
+        else:
+            comp_text = str(completion)
+        
+        if isinstance(prompt, list):
+            prompt_text = " ".join(m.get("content", "") for m in prompt)
+        else:
+            prompt_text = str(prompt)
+        
+        task = _classify_task_type(prompt_text)
+        
+        if task == "extraction":
+            rewards.append(reward_extraction(comp_text))
+        elif task == "sql_qa":
+            rewards.append(reward_sql_qa(comp_text))
+        elif task == "insights":
+            rewards.append(reward_insights(comp_text))
+        elif task == "push":
+            rewards.append(reward_push(comp_text))
+        else:
+            # Fallback: basic coherence
+            r = 0.2 if comp_text.strip() else 0.0
+            rewards.append(r)
+    
+    return rewards
+
+print("✓ Reward functions defined")
+```
+
+---
+
+## 7. Monitoring & Gate Conditions
+
+### Real-Time W&B Monitoring
+
+| Metric | Healthy Range | Stop Condition |
+|---|---|---|
+| `train/clip_ratio` | > 0 on majority of steps | Still 0 after step 20 on probe → abort |
+| `train/frac_reward_zero_std` | < 0.2 | Sustained > 0.5 → entropy collapse |
+| `train/reward` | Increasing trend, NOT starting at > 0.85 | Plateau at SFT-level → not learning |
+| `train/kl` | 0.01 – 0.5 | Near-zero → policy not moving; > 1.0 → instability |
+| `train/completion_length` | 50 – 400 | Hitting 512 ceiling → need to raise MAX_COMPLETION_LENGTH |
+| `eval/mean_reward` | Increasing trend | Plateau → early stopping will fire |
+
+### Success Criteria (Post-Training Validation)
+
+| Gate | Target | Pass/Fail |
+|---|---|---|
+| Extraction mean reward (20 samples) | ≥ 0.30 | Must pass |
+| Push mean reward (20 samples) | ≥ 0.40 | Must pass |
+| SQL Q&A mean reward (20 samples) | ≥ 0.20 | Should pass (lower bar — harder task for 0.5B) |
+| Insights mean reward (20 samples) | ≥ 0.20 | Should pass |
+| Overall mean > SFT calibration baseline | Mean V4 > Mean Cell 9 calibration | Must pass |
+
+---
+
+## 8. Fallback Plan
+
+### If Probe Gate Fails (clip_ratio = 0 on all 10 steps)
+
+**Step 1: Increase learning rate to 5e-6.** The model may need a stronger gradient push to overcome APO resistance.
+
+**Step 2: If still 0, try 0.5B-Base.** `Polygl0t/Tucano2-qwen-0.5B-Base` exists and has NO APO training. Load it, apply Unsloth LoRA, and repeat the probe. This requires NO SFT step — go directly Base → GRPO. The base model won't follow instructions well initially, but GRPO's reward signal should shape it.
+
+**Step 3: If Base also shows clip_ratio = 0, the issue is fundamental.** Possible causes: (a) TRL 0.24.0 bug in clip ratio computation, (b) reward function rewards are too uniform, (c) GRPO at this scale simply doesn't produce large enough probability changes per step. Try reducing `num_generations` to 8 (fewer completions = larger per-completion gradient contribution) and increasing `learning_rate` to 1e-5.
+
+**Step 4: If all above fail, switch to DPO.** Use the SFT model to generate completions, score them with reward functions, create preference pairs (chosen = highest reward, rejected = lowest reward in each group), and train iterative DPO. This bypasses the GRPO signal-to-noise issue entirely.
+
+### If Training Succeeds but Insights/SQL Scores Are < 0.15
+
+The 0.5B model may simply lack the capacity for analytical tasks. Accept this and plan the 3.7B scale-up for those tasks. Use the 0.5B results as validation that GRPO works on Tucano2-Instruct, then apply the validated recipe to `Polygl0t/Tucano2-qwen-3.7B-Instruct`.
+
+---
+
+## 9. Hyperparameter Decision Log
+
+| Parameter | Value | Rationale |
+|---|---|---|
+| model | `Polygl0t/Tucano2-qwen-0.5B-Instruct` | 2× better benchmarks than Think; no `<think>` overhead; structured output proven in model card |
+| temperature | 1.0 | Skywork-OR1 (2505.22312): τ=1.0 delays entropy collapse |
+| repetition_penalty | 1.0 (override from 1.2) | 1.2 suppresses diversity; GRPO needs maximally diverse rollouts |
+| num_generations | 16 | VRAM headroom at 0.5B allows G=16; more generations = more reward variance = stronger signal |
+| max_completion_length | 512 | No `<think>` overhead; extraction ~100 tok, SQL ~200, insights ~300 |
+| learning_rate | 2e-6 | Dr. GRPO Appendix G; 4× V2's 5e-7 to push harder against APO |
+| beta (KL) | 0.0 | Dr. GRPO §3.2: β=0 optimal for rule-based rewards; no ref model memory needed |
+| scale_rewards | False | Dr. GRPO: removes std normalization bias |
+| max_steps | 200 | Validation run; extend only if probe passes |
+| lora_r | 16 | Standard; matches V2/V3 SFT adapter |
+| lora_alpha | 32 | 2× lora_r |
+| batch_size | 2 | Effective batch: 2 prompts × 16 gen = 32 completions per step |
+| grad_accum | 1 | Keep effective batch small for faster iteration |
+| max_seq_length | 2048 | Model supports 4096; 2048 is generous for Instruct (no think overhead) |
+| use_cache | True (override from false) | Required for O(n) autoregressive generation |
+| top_k | 0 (override from 50) | Disable top-k; let temperature alone control diversity |
+
+---
+
+## 10. File Structure
+
+```
+tucano2_pipeline/
+├── v4_instruct_grpo.ipynb             ← THE NOTEBOOK (single model, all tasks)
+├── data/
+│   └── pairs/
+│       └── train.jsonl                ← existing full V2 training set (ALL tasks)
+└── models/
+    ├── tucano2-commerce-sft/          ← existing V2 SFT adapter (3.7B) — not used in V4
+    └── tucano2-0.5B-instruct-grpo-v4/ ← V4 output: Instruct model GRPO adapter
+```
+
+No task-specific data splits needed. No Think model artifacts.
+
+---
+
+*ADR-002 authored 2026-04-25. Based on direct audit of model repos `Polygl0t/Tucano2-qwen-0.5B-Instruct` and `Polygl0t/Tucano2-qwen-0.5B-Think`, cross-referenced with `docs/INVESTIGATION_REPORT.md` (20+ papers) and V1–V3 accumulated learnings.*