docs/ADR-002-v4-instruct.md

ADR-002: V4 Instruct-Only GRPO — Single-Model Validation on 0.5B-Instruct

Status: Proposed
Date: 2026-04-25
Author: Automated Investigation Agent
Supersedes: V4 Handoff (dual Instruct+Think plan)
Context Documents:

• docs/INVESTIGATION_REPORT.md — full project audit (20+ papers)
• docs/ADR-001-next-steps.md — V3 execution plans
• docs/checkpoints/2026-04-23_v3-launch.md — V3 launch & probe results
• V4 Handoff document (dual 0.5B hybrid plan)

---

Table of Contents

1. Context: Why the Original V4 Plan Needs Revision
2. Decisions: What We Do Instead and Why
3. Consequences: What to Expect
4. Verified Model Facts: Tokenizer & Architecture Reference
5. Implementation: Cell-by-Cell Notebook Specification
6. Reward Functions: Complete Specification
7. Monitoring & Gate Conditions
8. Fallback Plan: What If Instruct Fails
9. Hyperparameter Decision Log
10. File Structure

---

1. Context

1.1 V3 Autopsy (Confirmed)

V3 ran 171 steps on Polygl0t/Tucano2-qwen-3.7B-Think and failed:

train/reward:              0.8675   ← high but policy didn't learn; SFT already does this
train/clip_ratio:          0.0      ← zero on ALL 171 steps — policy never moved
train/kl:                  0.159    ← tiny divergence from initialization
train/completion_length:   2628     ← Think model fills context with <think>

The original V4 Handoff proposed a dual-model approach: train Tucano2-qwen-0.5B-Instruct for extraction+push, and Tucano2-qwen-0.5B-Think for SQL+insights, as two separate GRPO runs.

1.2 Problems Found in the Original V4 Plan

After auditing both model repos against their actual artifacts (config.json, generation_config.json, chat_template.jinja, training_config_sft.yaml, training_config_apo.yaml, tokenizer_config.json, README.md with benchmarks and inference samples), the following problems were identified:

Problem 1: The 0.5B-Think Model Is Catastrophically Weak

Published benchmarks from the model README:

Model	Total NPM	Knowledge & Reasoning NPM	GSM8K-PT	IFEval-PT
0.5B-Instruct	26.08	27.77	18.49	30.00
0.5B-Think	14.41	12.52	14.61	27.67

The Think variant scores nearly half of Instruct on every benchmark. The Think SFT used only 34M tokens of reasoning data for 3,060 steps, while Instruct SFT used 874M tokens across 9 task categories for 68,635 steps — that is 25× less data and 22× fewer steps for Think.

Concrete evidence from the Think model's own inference samples on the model card:

• Math (2x + 3 = 11): The Think model's CoT arrives at the wrong answer (x = 2 instead of x = 4). The model's thinking trace says "subtrairo 3 de ambos os lados" correctly, then in the final answer writes x = 5 - 3 = 2 — it contradicts its own reasoning.
• Cooking recipe: Hallucinates ingredient "30% ativo butylated buttercreme."
• History (Revolução Farroupilha): Fabricates dates, events, and named entities that don't exist.

By contrast, the Instruct model's inference samples show:

• Structured JSON output: Correct, well-formatted extraction from an email.
• Math: Correct solution (x = 4) without CoT.
• Function calling: Correct tool-use JSON.
• Classification: Correct sentiment classification.

Conclusion: A model that cannot solve 2x + 3 = 11 is not a viable starting point for SQL analysis or business insights, even with GRPO tuning. GRPO refines what a model approximately knows — it cannot teach fundamentally new capabilities to a model this weak.

Problem 2: Five Technical Bugs in the Original V4 Plan

Bug 2a: use_cache: false in both model configs.

Both config.json files ship with "use_cache": false. Without explicitly setting model.config.use_cache = True and model.generation_config.use_cache = True after loading, generation uses O(n²) full attention recomputation at every token. V3's notebook included this fix (Cell 4); the V4 plan omitted it entirely.

Source: Polygl0t/Tucano2-qwen-0.5B-Instruct/config.json, line "use_cache": false.

Bug 2b: repetition_penalty: 1.2 in generation_config.json.

Both models ship with repetition_penalty: 1.2. TRL's GRPOTrainer uses model.generate() internally for rollouts, and the generation_config.json defaults are loaded automatically. If repetition_penalty is not explicitly overridden to 1.0, it will suppress diversity in rollout completions — directly working against GRPO's need for diverse outputs. The GRPOConfig temperature parameter overrides the generation config's temperature, but there is no repetition_penalty field in GRPOConfig. It must be overridden via model.generation_config.repetition_penalty = 1.0 after model load.

Source: Polygl0t/Tucano2-qwen-0.5B-Instruct/generation_config.json, line "repetition_penalty": 1.2.

Bug 2c: temperature: 0.1 in generation_config.json.

Same as the V1 bug that destroyed the first training run. While GRPOConfig overrides temperature for rollouts, the model's generation_config may be used during eval callback generation if not explicitly overridden. Must set model.generation_config.temperature = 1.0 as a defensive measure.

Source: Polygl0t/Tucano2-qwen-0.5B-Instruct/generation_config.json, line "temperature": 0.1.

Bug 2d: Unsloth + tied word embeddings interaction.

Both 0.5B models have "tie_word_embeddings": true in config.json. When Unsloth applies LoRA, it targets linear projection layers. With tied embeddings, embed_tokens and lm_head share weights. If Unsloth's LoRA patching doesn't handle this correctly, gradients may not propagate to the output head, or the embedding table may drift independently. The smoke test (Cell 8) must verify that model.lm_head.weight.data_ptr() == model.model.embed_tokens.weight.data_ptr() holds after LoRA patching.

Source: Polygl0t/Tucano2-qwen-0.5B-Instruct/config.json, line "tie_word_embeddings": true.

Bug 2e: The V4 plan's reference policy VRAM estimate may be wrong.

The V4 VRAM budget includes "Reference policy (frozen copy) ~0.4GB." With beta=0.0 (no KL penalty), TRL 0.24.0's GRPOTrainer may skip loading the reference model entirely — the ref_model is only needed to compute KL divergence. But this behavior depends on the TRL version. If TRL loads the ref model anyway, it doubles the model footprint. This doesn't cause OOM at 0.5B (0.8GB total is fine), but it matters for the 3.7B scale-up. The smoke test must log peak VRAM to determine whether the ref model is loaded.

Problem 3: Hyperparameter Transfer From 0.5B to 3.7B Is Overstated

The original V4 plan claims hyperparameters validated at 0.5B transfer to 3.7B. This is partially true for qualitative findings (e.g., "clip_ratio > 0 is achievable," "task split works") but not for numerical values:

• LR=2e-6 at 0.5B (490M params) likely needs LR=5e-7 at 3.7B (3.8B params) — smaller models tolerate higher LR.
• G=16 at 0.5B is feasible with 512 completion length; at 3.7B the same VRAM budget supports G=4-8 at best.
• Effective batch size effects differ: batch=32 at 0.5B vs batch=8-16 at 3.7B changes gradient noise characteristics.

What transfers: the qualitative evidence that GRPO works on Tucano2-Instruct models, the reward function design, and the finding that APO-trained models can be further aligned (or can't).

---

2. Decisions

Decision 1: Single-Model, All-Task — 0.5B-Instruct Only

Decision: Train one GRPO run on Polygl0t/Tucano2-qwen-0.5B-Instruct using ALL four task types (extraction, push, sql_qa, insights). Do not train the 0.5B-Think model.

Rationale:

1. The Instruct model scores 26.08 NPM vs Think's 14.41 — nearly 2× better on every benchmark.
2. The Instruct model demonstrably produces correct structured JSON, correct math, correct function-call formatting (see model card samples).
3. The Instruct chat template does NOT inject <think> tokens. The assistant message is just {content} — clean output, no token budget conflict.
4. The Instruct model was SFT-trained on 874M tokens including structured output, retrieval, function calling, math with CoT, and general instruction following — it has a broad skill base suitable for all four tasks.
5. Running one model simplifies the notebook, eliminates the data-split complexity, and halves the compute budget.
6. If the Instruct model fails specifically on insights/analysis tasks, we can revisit Think for those tasks only. But the evidence says to test the strong model first.

Evidence: ThinkJSON (2502.14905) demonstrated that a 1.5B Instruct/Base model + GRPO beats DeepSeek-R1-671B on JSON extraction. The Instruct model doesn't need CoT to do structured output well. For analytical tasks, GRPO's reward signal can teach the model to produce structured analysis without explicit <think> overhead.

Decision 2: Use ALL Training Data (No Task Split)

Decision: Use the full V2 training set (data/pairs/train.jsonl, ~1,834 pairs) with the existing 40/40/10/10 distribution. Apply a 90/10 train/eval split. Do not create separate instruct/think data files.

Rationale:

1. More data = more diverse prompts = more GRPO signal. Splitting the data reduces each model's training set by ~50%.
2. Multi-task training at this scale is a feature, not a bug — the Cocktail Effect paper (2410.01109) shows mixing task types improves domain performance by 2-15%.
3. The reward function already dispatches by task type. GRPO handles mixed-task batches natively.

Decision 3: Override All Dangerous generation_config Defaults

Decision: After model load, explicitly override the following generation_config fields:

model.generation_config.temperature = 1.0
model.generation_config.repetition_penalty = 1.0
model.generation_config.do_sample = True
model.generation_config.top_k = 0         # disable top-k during GRPO rollouts
model.generation_config.top_p = 1.0       # disable top-p during GRPO rollouts
model.config.use_cache = True
model.generation_config.use_cache = True

Rationale:

• temperature=0.1 (default) destroyed V1. Must be overridden.
• repetition_penalty=1.2 (default) suppresses diversity. GRPO needs maximally diverse rollouts. Must be 1.0.
• top_k=50 and top_p=1.0 are set in the default generation_config. top_k=50 clips the distribution during sampling — at temp=1.0, this may unnecessarily restrict exploration. Set top_k=0 (disabled) to let temperature alone control diversity.
• use_cache=false (default) makes generation O(n²). Must be True.

Decision 4: Verify Tied Embeddings Survive LoRA Patching

Decision: Add a verification cell after model loading that checks:

# After Unsloth LoRA patching
assert model.lm_head.weight.data_ptr() == model.model.embed_tokens.weight.data_ptr(), \
    "CRITICAL: Tied embeddings broken after LoRA patching. lm_head and embed_tokens are now separate."

If this assertion fails, training may still work (gradients flow through LoRA layers on the projection matrices), but the embedding/output-head consistency that tie_word_embeddings=true provides would be broken. Document the result either way.

Decision 5: Hard Probe Gate on clip_ratio Before Full Training

Decision: Run a 10-step probe. If clip_ratio == 0.0 on all 10 steps, STOP. Do not proceed to full training. This was the missed signal in V3.

Gate condition: clip_ratio > 0.0 on at least 3 of 10 probe steps.

If the gate fails, proceed to Fallback Plan (Section 8) — do not iterate blindly.

Decision 6: Strip <think> Defensively in All Reward Functions

Decision: Even though the Instruct model's template doesn't inject <think>, the model may spontaneously generate think tokens (it has <think> as token ID 49116 in its vocabulary, and it was SFT-trained on math_cot data that contains reasoning traces). All reward functions must call strip_think() before scoring the answer portion.

---

3. Consequences

What We Expect

Metric	Expected Range	Justification
clip_ratio	> 0 on majority of steps	0.5B model has fewer params → larger per-param gradient; G=16 → more reward variance; no <think> overhead → shorter completions → less gradient dilution
Extraction reward	0.30 - 0.60	Instruct model already produces correct JSON (model card sample). GRPO refines schema compliance.
Push reward	0.40 - 0.70	Short outputs, Portuguese heuristics — simple task at any scale.
SQL Q&A reward	0.20 - 0.40	Model has general Portuguese comprehension. SQL-specific patterns need GRPO. Conservative target.
Insights reward	0.20 - 0.40	Model can follow instructions and structure output. Domain-specific vocabulary needs GRPO. Conservative target.
Completion length (Instruct)	50 - 300 tokens	No <think> overhead. Extraction ~100 tok, SQL ~200 tok, insights ~300 tok.
Training time	3 - 6 hours	0.5B is ~8× faster than 3.7B for generation. 200 steps × ~60-120s/step.

What This Validates for 3.7B Scale-Up

If V4 passes all gates:

1. GRPO works on APO-trained Tucano2 Instruct models. The APO anchor resistance hypothesis is disproven.
2. All-task training on a single model is viable. No need for a complex dual-model routing architecture.
3. Reward function calibration is confirmed. The same reward functions (with appropriate thresholds) can be used at 3.7B.
4. The winning recipe: 0.5B-Instruct + GRPO → scale to Polygl0t/Tucano2-qwen-3.7B-Instruct + GRPO.

What This Does NOT Validate

• Exact hyperparameter values (LR, G, completion length) for 3.7B.
• Whether 3.7B-Instruct has the same APO resistance characteristics as 0.5B-Instruct.
• Whether 3.7B fits in L4 VRAM at the same G and completion length.

---

4. Verified Model Facts

All values below were extracted directly from the actual repo files. The implementing agent should use these as ground truth, NOT the values from the original V4 handoff which contained some inaccuracies.

Tokenizer Token IDs (from tokenizer_config.json)

Token ID 0:     <|unk|>        (special=true)
Token ID 1:     <|im_start|>   (special=true)  — bos_token
Token ID 2:     <|im_end|>     (special=true)  — eos_token
Token ID 49109: <|pad|>        (special=true)  — pad_token
Token ID 49116: <think>        (special=false) — single token, NOT multi-token
Token ID 49117: </think>       (special=false) — single token, NOT multi-token
Token ID 49118: <answer>       (special=false)
Token ID 49119: </answer>      (special=false)

Critical note: <think> (49116) and </think> (49117) are registered as single dedicated tokens in added_tokens_decoder. The original V4 plan warned that </think> might be multi-token — this is WRONG. It is a single token. Two-pass generation using eos_token_id=[49117] to stop at </think> IS technically feasible. However, we do not need two-pass generation because we are using the Instruct model which does not generate <think> by default.

Model Architecture (from config.json)

model_type:              qwen3
architectures:           Qwen3ForCausalLM
num_hidden_layers:       28
hidden_size:             1024
intermediate_size:       3072
num_attention_heads:     16
num_key_value_heads:     8
head_dim:                128
vocab_size:              49152
max_position_embeddings: 4096
tie_word_embeddings:     true
use_cache:               false    ← MUST OVERRIDE TO true
rope_theta:              1000000
dtype:                   bfloat16
Parameters:              490,799,104

Generation Config (from generation_config.json — ALL must be overridden)

temperature:        0.1    ← OVERRIDE to 1.0 for training, 0.1 for eval
repetition_penalty: 1.2    ← OVERRIDE to 1.0 for training
do_sample:          true
max_new_tokens:     1024
eos_token_id:       [2]    (= <|im_end|>)

Instruct Chat Template Behavior (from chat_template.jinja)

The Instruct template applies the standard ChatML format:

<|im_start|>system
{system_content}<|im_end|>
<|im_start|>user
{user_content}<|im_end|>
<|im_start|>assistant
{assistant_content}<|im_end|>

With add_generation_prompt=True, the template appends <|im_start|>assistant\n to prompt the model to generate. There is no <think> injection anywhere in the Instruct template. The assistant block is rendered as plain {content} without any reasoning wrapper.

APO Training Details (from training_config_apo.yaml)

loss_type:            apo_zero
dpo_beta:             0.5
max_steps:            1115
max_learning_rate:    0.000005
num_train_epochs:     5
total_batch_size:     524288
reference_model:      Tucano2-qwen-0.5B-Instruct-SFT
precompute_ref_log_probs: true

The Instruct model had 1,115 steps of APO with dpo_beta=0.5. This is a moderate preference optimization — it creates a soft bias toward SFT behavior, not a hard constraint. With beta=0.0 in GRPO (no KL penalty) and LR=2e-6, the GRPO gradient should be strong enough to move the policy.

SFT Training Details (from training_config_sft.yaml)

Data: 874M tokens across 9 categories:
  - code:            ~2.3M tokens
  - function_call:   ~17.5M tokens
  - general:         ~700M tokens
  - math_cot:        ~27M tokens
  - retrieval:       ~2.2M tokens
  - structured:      ~35M tokens
  - summarization:   ~290K tokens
  - translation:     ~5.7M tokens
  - dpo (chosen):    ~14M tokens

max_steps:           68,635
max_learning_rate:   0.000085
assistant_only_loss: true

The model was trained on structured output (35M tokens) and function calling (17.5M tokens) — it has a strong foundation for extraction tasks.

---

5. Implementation: Cell-by-Cell Notebook Specification

The notebook is v4_instruct_grpo.ipynb. Each cell is a gate — verify output before proceeding.

Cell 1: Dependencies

# Cell 1 — Clean install
# Run after kernel restart

!pip install "unsloth"
!pip install "trl==0.24.0" --no-deps
!pip install "rich" "wandb"

Gate: No errors. Verify TRL 0.24.0 installed.

Cell 2: GPU + Unsloth Verification

import torch

print(f"CUDA available: {torch.cuda.is_available()}")
print(f"GPU: {torch.cuda.get_device_name(0)}")
print(f"VRAM: {torch.cuda.get_device_properties(0).total_mem / 1e9:.1f} GB")
print(f"bf16 support: {torch.cuda.is_bf16_supported()}")

from unsloth import FastLanguageModel
print(f"\n✓ Unsloth loaded")

import trl
assert trl.__version__ == "0.24.0", f"Expected TRL 0.24.0, got {trl.__version__}"
print(f"✓ TRL {trl.__version__}")

import transformers
print(f"✓ Transformers {transformers.__version__}")

Gate: CUDA available, bf16=True, VRAM > 20GB, TRL 0.24.0.

Cell 3: Config Constants

import os
import json
import re
import time
import random
from pathlib import Path

# ── Disable Unsloth kernel recompilation ─────────────────────────────────────
os.environ["UNSLOTH_COMPILE_DISABLE"] = "1"
os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True"

# ── Model ────────────────────────────────────────────────────────────────────
MODEL_ID       = "Polygl0t/Tucano2-qwen-0.5B-Instruct"
MAX_SEQ_LENGTH = 2048   # model supports 4096, but 2048 is plenty for Instruct (no <think> overhead)
ADAPTER_DIR    = Path("models/tucano2-0.5B-instruct-grpo-v4")
CHECKPOINT_DIR = ADAPTER_DIR / "checkpoints"

# ── Data ─────────────────────────────────────────────────────────────────────
DATA_DIR       = Path("data/pairs")
TRAIN_FILE     = DATA_DIR / "train.jsonl"
EVAL_SPLIT     = 0.10   # 10% held out for eval

# ── GRPO Hyperparameters ─────────────────────────────────────────────────────
NUM_GENERATIONS        = 16     # 0.5B + short completions = VRAM allows G=16
MAX_COMPLETION_LENGTH  = 512    # Instruct: no <think> overhead. Extraction ~100, SQL ~200, insights ~300
TEMPERATURE            = 1.0    # Skywork-OR1: τ=1.0 for exploration
LEARNING_RATE          = 2e-6   # Dr. GRPO: 4× V2's 5e-7 (clip_ratio=0 → push harder)
BETA                   = 0.0    # Dr. GRPO §3.2: β=0 optimal for rule-based rewards
SCALE_REWARDS          = False  # Dr. GRPO: remove std normalization bias
BATCH_SIZE             = 2      # per-device batch size
GRAD_ACCUM             = 1      # effective batch = 2 * 1 = 2 prompts * 16 gen = 32 completions
MAX_STEPS              = 200    # validation run
SAVE_STEPS             = 20
EVAL_STEPS             = 10
EARLY_STOPPING_PATIENCE = 15
EARLY_STOPPING_DELTA   = 0.005

# ── LoRA ─────────────────────────────────────────────────────────────────────
LORA_R     = 16
LORA_ALPHA = 32

# ── Monitoring ───────────────────────────────────────────────────────────────
WANDB_PROJECT     = "tucano2-commerce"
EVAL_MAX_SAMPLES  = 15    # eval callback samples
EVAL_MAX_TOKENS   = 512   # match training completion length

# ── Task Classification (inherited from V2/V3) ──────────────────────────────
VALID_SENTIMENTS  = {"positive", "negative", "neutral"}
VALID_CATEGORIES  = {
    "delivery_delay", "product_quality", "product_not_received",
    "wrong_product", "seller_communication", "app_issue",
    "price_value", "other", "none",
}
VALID_CHURN  = {"low", "medium", "high"}
VALID_REPEAT = {"yes", "no", "maybe"}
EXTRACTION_FIELDS = [
    "sentiment", "sentiment_score", "churn_risk", "delivery_issue",
    "product_issue", "seller_issue", "main_complaint",
    "complaint_category", "repeat_intent", "would_recommend",
]

# ── Verified Special Token IDs (from tokenizer_config.json) ─────────────────
# These are constants — do NOT recompute via tokenizer.encode()
TOKEN_ID_BOS       = 1      # <|im_start|>
TOKEN_ID_EOS       = 2      # <|im_end|>
TOKEN_ID_PAD       = 49109  # <|pad|>
TOKEN_ID_THINK     = 49116  # <think>
TOKEN_ID_THINK_END = 49117  # </think>

print("✓ Config loaded")
print(f"  Model: {MODEL_ID}")
print(f"  G={NUM_GENERATIONS}, max_comp={MAX_COMPLETION_LENGTH}, temp={TEMPERATURE}")
print(f"  LR={LEARNING_RATE}, β={BETA}, scale_rewards={SCALE_REWARDS}")
print(f"  LoRA r={LORA_R}, α={LORA_ALPHA}")
print(f"  Max steps: {MAX_STEPS}")

Cell 4: Load Model + Apply Critical Overrides

from unsloth import FastLanguageModel

print("Loading model...")
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name=MODEL_ID,
    max_seq_length=MAX_SEQ_LENGTH,
    load_in_4bit=True,
    dtype=None,  # auto-detect
)

# ═══════════════════════════════════════════════════════════════════════════════
# CRITICAL OVERRIDES — generation_config ships with values that destroy GRPO
# Source: Polygl0t/Tucano2-qwen-0.5B-Instruct/generation_config.json
#   temperature: 0.1       → override to 1.0
#   repetition_penalty: 1.2 → override to 1.0
#   use_cache: false        → override to true
# ═══════════════════════════════════════════════════════════════════════════════

model.config.use_cache = True
model.generation_config.use_cache = True
model.generation_config.temperature = TEMPERATURE
model.generation_config.repetition_penalty = 1.0   # CRITICAL: 1.2 suppresses diversity
model.generation_config.do_sample = True
model.generation_config.top_k = 0    # disable top-k — let temperature control diversity
model.generation_config.top_p = 1.0  # disable top-p

# Pad token
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

print(f"✓ Model loaded on {model.device}")
print(f"  use_cache: {model.config.use_cache}")
print(f"  temperature: {model.generation_config.temperature}")
print(f"  repetition_penalty: {model.generation_config.repetition_penalty}")
print(f"  top_k: {model.generation_config.top_k}")
print(f"  Params: {sum(p.numel() for p in model.parameters()) / 1e6:.0f}M")

# ═══════════════════════════════════════════════════════════════════════════════
# TIED EMBEDDINGS CHECK
# Source: config.json has "tie_word_embeddings": true
# If Unsloth LoRA patching breaks this, log it (may not be fatal).
# ═══════════════════════════════════════════════════════════════════════════════

try:
    lm_ptr = model.lm_head.weight.data_ptr()
    embed_ptr = model.model.embed_tokens.weight.data_ptr()
    tied = lm_ptr == embed_ptr
    print(f"  Tied embeddings intact: {tied}")
    if not tied:
        print("  ⚠️ WARNING: Tied embeddings broken after Unsloth load. May affect output head gradients.")
except AttributeError as e:
    print(f"  ⚠️ Could not check tied embeddings: {e}")

Gate: Model loaded, use_cache=True, repetition_penalty=1.0, temperature=1.0.

Cell 5: Token ID Verification

# Verify that the constants from Cell 3 match the actual tokenizer
# Do NOT skip this cell — if IDs don't match, all reward functions break

tok_tests = {
    "<|im_start|>": TOKEN_ID_BOS,
    "<|im_end|>":   TOKEN_ID_EOS,
    "<|pad|>":      TOKEN_ID_PAD,
    "<think>":      TOKEN_ID_THINK,
    "</think>":     TOKEN_ID_THINK_END,
}

all_pass = True
for text, expected_id in tok_tests.items():
    # For special tokens registered in added_tokens, encode should return single ID
    ids = tokenizer.encode(text, add_special_tokens=False)
    actual_id = ids[0] if len(ids) == 1 else ids
    match = (len(ids) == 1 and ids[0] == expected_id)
    status = "✓" if match else "✗"
    print(f"  {status} '{text}' → expected {expected_id}, got {actual_id}")
    if not match:
        all_pass = False

assert all_pass, "Token ID mismatch detected. Update constants in Cell 3 before proceeding."
print("\n✓ All token IDs verified")

# Also verify eos_token_id is correct
assert tokenizer.eos_token_id == TOKEN_ID_EOS, f"eos_token_id mismatch: {tokenizer.eos_token_id}"
print(f"✓ eos_token_id = {tokenizer.eos_token_id}")

Gate: All token IDs match. Single-token <think> (49116) and </think> (49117) confirmed.

Cell 6: KV Cache Diagnostic

# Copied from V2 Cell 5b — verify KV cache is working
# Gate: ratio < 3× → KV cache OK. ratio > 5× → BROKEN, abort.

FastLanguageModel.for_inference(model)

_kv_msgs = [{"role": "user", "content": "Qual a categoria de reclamação mais frequente?"}]
_kv_text = tokenizer.apply_chat_template(_kv_msgs, tokenize=False, add_generation_prompt=True)
_kv_inputs = tokenizer(_kv_text, return_tensors="pt").to(model.device)

_token_times, _past, _generated = [], None, _kv_inputs["input_ids"]
with torch.no_grad():
    for _step in range(50):
        _t0 = time.time()
        seq_len = _generated.shape[1]
        if _past is None:
            _position_ids = torch.arange(seq_len, dtype=torch.long, device=model.device).unsqueeze(0)
        else:
            _position_ids = torch.tensor([[seq_len - 1]], dtype=torch.long, device=model.device)
        _out = model(
            input_ids=_generated[:, -1:] if _past else _generated,
            position_ids=_position_ids,
            attention_mask=torch.ones(1, seq_len, device=model.device),
            past_key_values=_past,
            use_cache=True,
            return_dict=True,
        )
        _past = _out.past_key_values
        _next = _out.logits[:, -1, :].argmax(dim=-1, keepdim=True)
        _generated = torch.cat([_generated, _next], dim=1)
        _token_times.append(time.time() - _t0)

_ratio = sum(_token_times[45:]) / max(sum(_token_times[:5]), 1e-9)
print(f"First 5 tok: {[f'{t*1000:.0f}ms' for t in _token_times[:5]]}")
print(f"Last  5 tok: {[f'{t*1000:.0f}ms' for t in _token_times[45:]]}")
print(f"Ratio last/first: {_ratio:.1f}x")
assert _ratio < 5, f"KV cache BROKEN (ratio {_ratio:.1f}×). Check model.config.use_cache."
print("✓ KV cache working correctly")

del _past, _generated, _kv_inputs, _token_times, _out
import gc; gc.collect()
torch.cuda.empty_cache()

Gate: Ratio < 3×.

Cell 7: Single Inference Test

# Verify model generates coherent Portuguese and closes <|im_end|>

FastLanguageModel.for_inference(model)

test_msgs = [
    {"role": "system", "content": "Você é um assistente de IA especializado em e-commerce brasileiro."},
    {"role": "user", "content": "Analise esta avaliação: 'Produto chegou quebrado, péssima embalagem. Nunca mais compro aqui.' Retorne um objeto JSON com os campos: sentiment, sentiment_score, delivery_issue, complaint_category."},
]
text = tokenizer.apply_chat_template(test_msgs, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

t0 = time.time()
outputs = model.generate(
    **inputs,
    max_new_tokens=256,
    temperature=0.1,       # low temp for deterministic eval
    do_sample=True,
    repetition_penalty=1.0,
)
elapsed = time.time() - t0

response = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(f"Generation time: {elapsed:.1f}s")
print(f"Response length: {len(response)} chars")
print(f"Contains <think>: {'<think>' in response}")
print(f"Contains JSON {{ }}: {'{' in response and '}' in response}")
print(f"\n{'='*60}")
print(response[:500])

Gate: Response is coherent Portuguese. Check whether <think> appears (document the result — this tells us if the Instruct model spontaneously thinks). Check if JSON structure is present.

Cell 8: Reward Functions

Complete reward functions — see Section 6 below for the full specification. This cell defines:

• strip_think(text) — remove <think>...</think> blocks
• has_think_block(text) — check for think blocks
• _classify_task_type(prompt_text) — classify prompt into task type
• _extract_json(text) — extract JSON from text robustly
• reward_extraction(completion) — continuous reward for JSON extraction (max 1.0)
• reward_sql_qa(completion) — continuous reward for SQL Q&A (max 1.0)
• reward_insights(completion) — continuous reward for insights (max 1.0)
• reward_push(completion) — continuous reward for push notifications (max 1.0)
• commerce_reward_fn(completions, prompts, **kwargs) — master dispatch function

Cell 9: Reward Calibration

# Load data, classify by task type, run calibration on 8 diverse samples

by_type = {"extraction": [], "sql_qa": [], "insights": [], "push": []}
with open(TRAIN_FILE) as f:
    for line in f:
        row = json.loads(line)
        convs = row["conversations"]
        prompt_msgs = [m for m in convs if m["role"] in ("system", "user")]
        if not prompt_msgs:
            continue
        user_text = " ".join(m["content"] for m in prompt_msgs if m["role"] == "user")
        task = _classify_task_type(user_text)
        by_type[task].append(prompt_msgs)

print(f"Prompts by type: {', '.join(f'{k}={len(v)}' for k, v in by_type.items())}")

# Pick 2 samples per task type = 8 total
rng = random.Random(42)
cal_samples = []
for task_type in by_type:
    pool = by_type[task_type]
    if len(pool) >= 2:
        cal_samples.extend(rng.sample(pool, 2))
    elif pool:
        cal_samples.extend(pool)

FastLanguageModel.for_inference(model)
print(f"\nReward calibration ({len(cal_samples)} samples):")
print("-" * 60)

cal_rewards = []
for i, msgs in enumerate(cal_samples):
    text = tokenizer.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
    inputs = tokenizer(text, return_tensors="pt").to(model.device)
    outputs = model.generate(
        **inputs,
        max_new_tokens=MAX_COMPLETION_LENGTH,
        temperature=0.7,
        do_sample=True,
        repetition_penalty=1.0,
    )
    response = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
    r = commerce_reward_fn([response], [text])[0]
    cal_rewards.append(r)
    task = _classify_task_type(" ".join(m.get("content", "") for m in msgs if m["role"] == "user"))
    has_think = "<think>" in response
    answer_preview = strip_think(response)[:100]
    print(f"  Sample {i+1} [{task:12s}]: reward={r:.2f} | has_think={has_think} | {answer_preview}")

print(f"\nMean={sum(cal_rewards)/len(cal_rewards):.2f}, Min={min(cal_rewards):.2f}, Max={max(cal_rewards):.2f}")
print(f"Reward variance > 0: {len(set(f'{r:.4f}' for r in cal_rewards)) > 1}")

Gate: Mean reward < 0.90 (if already ~1.0, the reward function is too easy — GRPO won't learn). Variance > 0. Document whether <think> appeared.

Cell 10: Dataset Preparation

from datasets import Dataset

def prepare_datasets(train_file, eval_ratio=EVAL_SPLIT, seed=42):
    rng = random.Random(seed)
    
    all_records = []
    with open(train_file) as f:
        for line in f:
            row = json.loads(line)
            convs = row["conversations"]
            prompt_msgs = [m for m in convs if m["role"] in ("system", "user")]
            if prompt_msgs:
                all_records.append(prompt_msgs)
    
    rng.shuffle(all_records)
    n_eval = max(1, int(len(all_records) * eval_ratio))
    eval_records = all_records[:n_eval]
    train_records = all_records[n_eval:]
    
    # Log task distribution
    for label, records in [("train", train_records), ("eval", eval_records)]:
        dist = {}
        for msgs in records:
            user_text = " ".join(m["content"] for m in msgs if m["role"] == "user")
            task = _classify_task_type(user_text)
            dist[task] = dist.get(task, 0) + 1
        print(f"  {label}: {len(records)} prompts — {dist}")
    
    train_ds = Dataset.from_list([{"prompt": msgs} for msgs in train_records])
    eval_ds = Dataset.from_list([{"prompt": msgs} for msgs in eval_records])
    return train_ds, eval_ds

train_dataset, eval_dataset = prepare_datasets(TRAIN_FILE)
print(f"\n✓ Datasets: train={len(train_dataset)}, eval={len(eval_dataset)}")

Gate: Train has ~1,650 prompts, eval has ~180. All 4 task types present in both.

Cell 11: Smoke Test (1 Step)

from trl import GRPOConfig, GRPOTrainer

FastLanguageModel.for_training(model)

smoke_config = GRPOConfig(
    output_dir=str(CHECKPOINT_DIR / "smoke"),
    num_generations=NUM_GENERATIONS,
    scale_rewards=SCALE_REWARDS,
    max_completion_length=MAX_COMPLETION_LENGTH,
    max_steps=1,
    temperature=TEMPERATURE,
    per_device_train_batch_size=BATCH_SIZE,
    gradient_accumulation_steps=1,
    learning_rate=LEARNING_RATE,
    fp16=False,
    bf16=True,
    logging_steps=1,
    save_steps=999,
    report_to="none",
    max_prompt_length=MAX_SEQ_LENGTH // 2,
    seed=42,
    remove_unused_columns=False,
)

# ── UnslothGRPOTrainer (inherited from V2/V3) ────────────────────────────────
class UnslothGRPOTrainer(GRPOTrainer):
    def _generate(self, prompts, images):
        FastLanguageModel.for_inference(self.model)
        try:
            result = super()._generate(prompts, images)
        finally:
            FastLanguageModel.for_training(self.model)
        return result

smoke_trainer = UnslothGRPOTrainer(
    model=model,
    reward_funcs=commerce_reward_fn,
    args=smoke_config,
    train_dataset=train_dataset,
    processing_class=tokenizer,
)

t0 = time.time()
smoke_trainer.train()
step_time = time.time() - t0

peak_vram = torch.cuda.max_memory_allocated() / 1e9
print(f"\n✓ Smoke test passed!")
print(f"  Step time: {step_time:.0f}s")
print(f"  Peak VRAM: {peak_vram:.1f}GB / {torch.cuda.get_device_properties(0).total_mem / 1e9:.1f}GB")
print(f"  Estimated full run ({MAX_STEPS} steps): {step_time * MAX_STEPS / 3600:.1f}h")

del smoke_trainer
gc.collect(); torch.cuda.empty_cache()

Gate: No OOM. Peak VRAM < 20GB. Step time < 180s. Document whether ref model was loaded (check VRAM: if peak > 1.0GB, ref model is loaded; if ~0.5GB, it's skipped due to β=0).

Cell 12: Probe Run (10 Steps) — THE CRITICAL GATE

FastLanguageModel.for_training(model)

probe_config = GRPOConfig(
    output_dir=str(CHECKPOINT_DIR / "probe"),
    num_generations=NUM_GENERATIONS,
    scale_rewards=SCALE_REWARDS,
    max_completion_length=MAX_COMPLETION_LENGTH,
    max_steps=10,
    temperature=TEMPERATURE,
    num_train_epochs=1,
    per_device_train_batch_size=BATCH_SIZE,
    gradient_accumulation_steps=GRAD_ACCUM,
    learning_rate=LEARNING_RATE,
    warmup_ratio=0.1,
    lr_scheduler_type="cosine",
    fp16=False,
    bf16=True,
    logging_steps=1,
    save_steps=999,
    report_to="none",
    max_prompt_length=MAX_SEQ_LENGTH // 2,
    seed=42,
    remove_unused_columns=False,
)

probe_trainer = UnslothGRPOTrainer(
    model=model,
    reward_funcs=commerce_reward_fn,
    args=probe_config,
    train_dataset=train_dataset,
    processing_class=tokenizer,
)

t0 = time.time()
result = probe_trainer.train()
elapsed = time.time() - t0

# ══════════════════════════════════════════════════════════════════════════════
# CRITICAL GATE: clip_ratio > 0 on at least 3 of 10 steps
# If this fails, STOP. See Fallback Plan (Section 8 of ADR-002).
# ══════════════════════════════════════════════════════════════════════════════
# TRL logs clip_ratio in training history. Extract from trainer.state.log_history.
clip_ratios = []
for entry in probe_trainer.state.log_history:
    if "train/clip_ratio" in entry:
        clip_ratios.append(entry["train/clip_ratio"])

nonzero_clips = sum(1 for cr in clip_ratios if cr > 0.0)
print(f"\n{'='*60}")
print(f"PROBE RESULTS ({elapsed:.0f}s, {elapsed/10:.0f}s/step)")
print(f"  clip_ratios: {[f'{cr:.4f}' for cr in clip_ratios]}")
print(f"  Non-zero clip steps: {nonzero_clips}/{len(clip_ratios)}")
print(f"  Train loss: {result.training_loss:.4f}")
print(f"{'='*60}")

if nonzero_clips >= 3:
    print("✓ PROBE GATE PASSED — proceed to full training")
elif nonzero_clips > 0:
    print("⚠️ MARGINAL — clip_ratio > 0 on some steps but < 3. Consider increasing LR or G.")
else:
    print("✗ PROBE GATE FAILED — clip_ratio = 0 on ALL steps.")
    print("  DO NOT proceed to full training.")
    print("  See ADR-002 Section 8 (Fallback Plan).")

del probe_trainer
gc.collect(); torch.cuda.empty_cache()

Gate: nonzero_clips >= 3. If this fails, go to Section 8.

Cell 13: W&B Init + Full Training

import wandb

wandb.login()
wandb.init(
    project=WANDB_PROJECT,
    name=f"grpo-v4-instruct-0.5B-{time.strftime('%Y%m%d-%H%M')}",
    config={
        "model_id": MODEL_ID,
        "version": "v4",
        "num_generations": NUM_GENERATIONS,
        "max_completion_length": MAX_COMPLETION_LENGTH,
        "temperature": TEMPERATURE,
        "learning_rate": LEARNING_RATE,
        "beta": BETA,
        "scale_rewards": SCALE_REWARDS,
        "batch_size": BATCH_SIZE,
        "grad_accum": GRAD_ACCUM,
        "max_steps": MAX_STEPS,
        "lora_r": LORA_R,
        "lora_alpha": LORA_ALPHA,
        "train_prompts": len(train_dataset),
        "eval_prompts": len(eval_dataset),
        "repetition_penalty_override": 1.0,
    },
)
print(f"✓ W&B run: {wandb.run.url}")

# ── EvalRewardCallback (inherited from V2/V3, adapted) ──────────────────────
from transformers import TrainerCallback

class EvalRewardCallback(TrainerCallback):
    def __init__(self, eval_records, reward_fn, patience, delta):
        self.eval_records = eval_records
        self.reward_fn = reward_fn
        self.patience = patience
        self.delta = delta
        self.best_reward = -float("inf")
        self.best_step = 0
        self.no_improve_count = 0

    def on_step_end(self, args, state, control, model=None, processing_class=None, **kwargs):
        if state.global_step == 0 or state.global_step % EVAL_STEPS != 0:
            return control

        tokenizer_local = processing_class
        if tokenizer_local is None:
            print("[EvalRewardCallback] WARNING: tokenizer is None, skipping eval")
            return control

        mean_reward = self._run_eval(model, tokenizer_local, args)
        improved = mean_reward > self.best_reward + self.delta

        wandb.log({
            "eval/mean_reward": mean_reward,
            "eval/best_reward": max(self.best_reward, mean_reward),
            "eval/no_improve_count": self.no_improve_count,
        }, step=state.global_step)

        status = "↑ improved" if improved else f"↔ no gain ({self.no_improve_count + 1}/{self.patience})"
        print(f"\n[EvalReward] step={state.global_step} | mean={mean_reward:.4f} | best={self.best_reward:.4f} | {status}")

        if improved:
            self.best_reward = mean_reward
            self.best_step = state.global_step
            self.no_improve_count = 0
        else:
            self.no_improve_count += 1
            if self.no_improve_count >= self.patience:
                print(f"[EarlyStopping] No improvement for {self.patience} evals. Halting.")
                control.should_training_stop = True
        return control

    def _run_eval(self, model, tokenizer_local, args):
        FastLanguageModel.for_inference(model)
        rewards = []
        subset = self.eval_records[:EVAL_MAX_SAMPLES]
        for record in subset:
            msgs = record["prompt"]
            text = tokenizer_local.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
            inputs = tokenizer_local(text, return_tensors="pt", truncation=True, max_length=args.max_prompt_length).to(model.device)
            with torch.no_grad():
                out = model.generate(
                    **inputs,
                    max_new_tokens=EVAL_MAX_TOKENS,
                    temperature=0.1,   # deterministic eval
                    do_sample=True,
                    repetition_penalty=1.0,
                )
            resp = tokenizer_local.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
            rewards.append(self.reward_fn([resp], [text])[0])
        FastLanguageModel.for_training(model)
        return sum(rewards) / len(rewards) if rewards else 0.0

# ── Training ────────────────────────────────────────────────────────────────
FastLanguageModel.for_training(model)

grpo_config = GRPOConfig(
    output_dir=str(CHECKPOINT_DIR),
    num_generations=NUM_GENERATIONS,
    scale_rewards=SCALE_REWARDS,
    max_completion_length=MAX_COMPLETION_LENGTH,
    max_steps=MAX_STEPS,
    temperature=TEMPERATURE,
    num_train_epochs=1,
    per_device_train_batch_size=BATCH_SIZE,
    gradient_accumulation_steps=GRAD_ACCUM,
    learning_rate=LEARNING_RATE,
    warmup_ratio=0.1,
    lr_scheduler_type="cosine",
    fp16=False,
    bf16=True,
    logging_steps=1,
    save_steps=SAVE_STEPS,
    save_total_limit=5,
    save_only_model=True,
    report_to="wandb",
    max_prompt_length=MAX_SEQ_LENGTH // 2,
    seed=42,
    remove_unused_columns=False,
    disable_tqdm=True,
    logging_first_step=True,
)

eval_cb = EvalRewardCallback(
    eval_records=list(eval_dataset),
    reward_fn=commerce_reward_fn,
    patience=EARLY_STOPPING_PATIENCE,
    delta=EARLY_STOPPING_DELTA,
)

trainer = UnslothGRPOTrainer(
    model=model,
    reward_funcs=commerce_reward_fn,
    args=grpo_config,
    train_dataset=train_dataset,
    processing_class=tokenizer,
    callbacks=[eval_cb],
)

t_start = time.time()
result = trainer.train()
elapsed = time.time() - t_start

wandb.log({
    "train/final_loss": result.training_loss,
    "train/duration_hours": elapsed / 3600,
    "train/total_steps": result.global_step,
    "eval/best_reward_final": eval_cb.best_reward,
    "eval/best_step": eval_cb.best_step,
})
wandb.finish()

print(f"\n{'='*60}")
print(f"V4 Training Complete")
print(f"  Loss:        {result.training_loss:.4f}")
print(f"  Steps:       {result.global_step}")
print(f"  Duration:    {elapsed/3600:.1f}h")
print(f"  Best eval:   {eval_cb.best_reward:.4f} (step {eval_cb.best_step})")
print(f"{'='*60}")

Cell 14: Validation (20 Held-Out Samples)

# Run validation on 20 held-out samples, broken down by task type

FastLanguageModel.for_inference(model)

val_samples = list(eval_dataset)[:20]
val_results = {"extraction": [], "sql_qa": [], "insights": [], "push": []}

for i, record in enumerate(val_samples):
    msgs = record["prompt"]
    user_text = " ".join(m["content"] for m in msgs if m["role"] == "user")
    task = _classify_task_type(user_text)
    
    text = tokenizer.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
    inputs = tokenizer(text, return_tensors="pt").to(model.device)
    with torch.no_grad():
        out = model.generate(
            **inputs,
            max_new_tokens=MAX_COMPLETION_LENGTH,
            temperature=0.1,
            do_sample=True,
            repetition_penalty=1.0,
        )
    resp = tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
    r = commerce_reward_fn([resp], [text])[0]
    val_results[task].append(r)
    print(f"  [{task:12s}] reward={r:.2f} | {strip_think(resp)[:80]}")

print(f"\n{'='*60}")
print("Validation Results by Task:")
for task, rewards in val_results.items():
    if rewards:
        mean_r = sum(rewards) / len(rewards)
        print(f"  {task:12s}: mean={mean_r:.3f} (n={len(rewards)})")
print(f"{'='*60}")

Cell 15: Save Adapter

# Save the GRPO-tuned LoRA adapter

model.save_pretrained(str(ADAPTER_DIR))
tokenizer.save_pretrained(str(ADAPTER_DIR))
print(f"✓ Adapter saved to {ADAPTER_DIR}")

---

6. Reward Functions: Complete Specification

These are the exact reward functions the implementing agent must use. They are adapted from V2/V3 with one critical change: strip_think() is called defensively on ALL completions before scoring, even for the Instruct model.

def strip_think(text: str) -> str:
    """Remove <think>...</think> block, return the answer portion."""
    return re.sub(r"<think>.*?</think>", "", text, flags=re.DOTALL).strip()

def has_think_block(text: str) -> bool:
    return bool(re.search(r"<think>.+</think>", text, flags=re.DOTALL))

def _classify_task_type(prompt_text: str) -> str:
    p = prompt_text.lower()
    if "retorne um objeto json" in p or "extraia dados" in p or "json" in p:
        return "extraction"
    elif "notificação push" in p or "notificação de reengajamento" in p:
        return "push"
    elif "perfil do cliente" in p or "retenção" in p or "análise" in p or "insight" in p:
        return "insights"
    else:
        return "sql_qa"

def _extract_json(text: str) -> dict | None:
    """Extract first JSON object from text. Returns parsed dict or None."""
    # Try direct parse first
    stripped = text.strip()
    # Remove markdown code blocks if present
    stripped = re.sub(r"^```(?:json)?\s*", "", stripped)
    stripped = re.sub(r"\s*```$", "", stripped)
    stripped = stripped.strip()
    try:
        return json.loads(stripped)
    except (json.JSONDecodeError, TypeError):
        pass
    # Try to find JSON object within text
    match = re.search(r"\{[^{}]*(?:\{[^{}]*\}[^{}]*)*\}", text, re.DOTALL)
    if match:
        try:
            return json.loads(match.group())
        except (json.JSONDecodeError, TypeError):
            pass
    return None

def reward_extraction(completion: str) -> float:
    """Continuous reward for extraction tasks (max 1.0)."""
    answer = strip_think(completion)
    data = _extract_json(answer)
    
    if data is None:
        # Partial credit for JSON-like structure
        if "{" in answer and "}" in answer:
            return 0.05
        return 0.0
    
    if not isinstance(data, dict):
        return 0.1  # valid JSON but not an object
    
    score = 0.3  # valid JSON object
    
    # Schema completeness (0.3 total)
    present = sum(1 for f in EXTRACTION_FIELDS if f in data)
    score += 0.3 * (present / len(EXTRACTION_FIELDS))
    
    # Value validity (0.4 total, split across checks)
    checks_passed = 0
    checks_total = 0
    
    for field, validator in [
        ("sentiment", lambda v: v in VALID_SENTIMENTS),
        ("complaint_category", lambda v: v in VALID_CATEGORIES),
        ("churn_risk", lambda v: v in VALID_CHURN),
        ("repeat_intent", lambda v: v in VALID_REPEAT),
        ("sentiment_score", lambda v: isinstance(v, (int, float)) and 1 <= v <= 5),
    ]:
        checks_total += 1
        if field in data and validator(data[field]):
            checks_passed += 1
    
    for bool_field in ("delivery_issue", "product_issue", "seller_issue", "would_recommend"):
        checks_total += 1
        if bool_field in data and isinstance(data[bool_field], bool):
            checks_passed += 1
    
    if checks_total > 0:
        score += 0.4 * (checks_passed / checks_total)
    
    return min(score, 1.0)

def reward_sql_qa(completion: str) -> float:
    """Continuous reward for SQL Q&A (max 1.0)."""
    answer = strip_think(completion)
    if not answer.strip():
        return 0.0
    
    score = 0.0
    
    # Numerical content (more numbers = more specific answer)
    numbers = re.findall(r"\d+(?:[.,]\d+)?", answer)
    score += min(0.4, 0.1 * len(numbers))
    
    # Length: 50-500 chars optimal
    length = len(answer)
    if 50 <= length <= 500:
        score += 0.3
    elif length > 0:
        score += 0.3 * max(0, 1 - abs(length - 275) / 275)
    
    # Portuguese business vocabulary
    pt_business = ["pedidos", "clientes", "média", "total", "taxa", "vendas",
                   "produtos", "período", "categoria", "região", "faturamento"]
    pt_matches = sum(1 for w in pt_business if w in answer.lower())
    score += min(0.3, 0.06 * pt_matches)
    
    return min(score, 1.0)

def reward_insights(completion: str) -> float:
    """Continuous reward for insights (max 1.0)."""
    answer = strip_think(completion)
    if not answer.strip():
        return 0.0
    
    score = 0.0
    
    # Actionable language
    action_words = ["recomend", "implement", "melhor", "reduzir", "aumentar",
                    "priorizar", "investir", "otimizar", "estratégi", "ação"]
    matches = sum(1 for w in action_words if w in answer.lower())
    score += min(0.4, 0.08 * matches)
    
    # Length: 100-800 chars optimal
    length = len(answer)
    if 100 <= length <= 800:
        score += 0.3
    elif length > 0:
        score += 0.3 * max(0, 1 - abs(length - 450) / 450)
    
    # Structure: bullet points, numbered lists, headers
    structure_marks = len(re.findall(r"^[-•*]\s|^\d+[.)]\s|^#{1,3}\s", answer, re.MULTILINE))
    score += min(0.2, 0.04 * structure_marks)
    
    # Portuguese coherence marker
    if any(w in answer.lower() for w in ["cliente", "produto", "serviço", "empresa"]):
        score += 0.1
    
    return min(score, 1.0)

def reward_push(completion: str) -> float:
    """Continuous reward for push notifications (max 1.0)."""
    answer = strip_think(completion).strip()
    if not answer:
        return 0.0
    
    # Length: ≤120 chars gets full credit
    length = len(answer)
    if length <= 120:
        length_score = 0.5
    else:
        length_score = 0.5 * max(0, 1 - (length - 120) / 120)
    
    # Portuguese content
    pt_markers = re.findall(r"[ãçéêóúâõ]|você|para|como|seu|sua|oferta|desconto|produto",
                            answer, re.IGNORECASE)
    lang_score = min(0.3, 0.03 * len(pt_markers))
    
    # Non-generic (penalize very generic phrases)
    generic = ["olá", "obrigado pela compra", "agradecemos"]
    is_generic = any(g in answer.lower() for g in generic)
    creativity_score = 0.0 if is_generic else 0.2
    
    return min(length_score + lang_score + creativity_score, 1.0)

def commerce_reward_fn(completions, prompts, **kwargs) -> list[float]:
    """Master reward function: dispatches by task type."""
    rewards = []
    for completion, prompt in zip(completions, prompts):
        if isinstance(completion, list):
            comp_text = completion[-1]["content"] if completion else ""
        else:
            comp_text = str(completion)
        
        if isinstance(prompt, list):
            prompt_text = " ".join(m.get("content", "") for m in prompt)
        else:
            prompt_text = str(prompt)
        
        task = _classify_task_type(prompt_text)
        
        if task == "extraction":
            rewards.append(reward_extraction(comp_text))
        elif task == "sql_qa":
            rewards.append(reward_sql_qa(comp_text))
        elif task == "insights":
            rewards.append(reward_insights(comp_text))
        elif task == "push":
            rewards.append(reward_push(comp_text))
        else:
            # Fallback: basic coherence
            r = 0.2 if comp_text.strip() else 0.0
            rewards.append(r)
    
    return rewards

print("✓ Reward functions defined")

---

7. Monitoring & Gate Conditions

Real-Time W&B Monitoring

Metric	Healthy Range	Stop Condition
train/clip_ratio	> 0 on majority of steps	Still 0 after step 20 on probe → abort
train/frac_reward_zero_std	< 0.2	Sustained > 0.5 → entropy collapse
train/reward	Increasing trend, NOT starting at > 0.85	Plateau at SFT-level → not learning
train/kl	0.01 – 0.5	Near-zero → policy not moving; > 1.0 → instability
train/completion_length	50 – 400	Hitting 512 ceiling → need to raise MAX_COMPLETION_LENGTH
eval/mean_reward	Increasing trend	Plateau → early stopping will fire

Success Criteria (Post-Training Validation)

Gate	Target	Pass/Fail
Extraction mean reward (20 samples)	≥ 0.30	Must pass
Push mean reward (20 samples)	≥ 0.40	Must pass
SQL Q&A mean reward (20 samples)	≥ 0.20	Should pass (lower bar — harder task for 0.5B)
Insights mean reward (20 samples)	≥ 0.20	Should pass
Overall mean > SFT calibration baseline	Mean V4 > Mean Cell 9 calibration	Must pass

---

8. Fallback Plan

If Probe Gate Fails (clip_ratio = 0 on all 10 steps)

Step 1: Increase learning rate to 5e-6. The model may need a stronger gradient push to overcome APO resistance.

Step 2: If still 0, try 0.5B-Base. Polygl0t/Tucano2-qwen-0.5B-Base exists and has NO APO training. Load it, apply Unsloth LoRA, and repeat the probe. This requires NO SFT step — go directly Base → GRPO. The base model won't follow instructions well initially, but GRPO's reward signal should shape it.

Step 3: If Base also shows clip_ratio = 0, the issue is fundamental. Possible causes: (a) TRL 0.24.0 bug in clip ratio computation, (b) reward function rewards are too uniform, (c) GRPO at this scale simply doesn't produce large enough probability changes per step. Try reducing num_generations to 8 (fewer completions = larger per-completion gradient contribution) and increasing learning_rate to 1e-5.

Step 4: If all above fail, switch to DPO. Use the SFT model to generate completions, score them with reward functions, create preference pairs (chosen = highest reward, rejected = lowest reward in each group), and train iterative DPO. This bypasses the GRPO signal-to-noise issue entirely.

If Training Succeeds but Insights/SQL Scores Are < 0.15

The 0.5B model may simply lack the capacity for analytical tasks. Accept this and plan the 3.7B scale-up for those tasks. Use the 0.5B results as validation that GRPO works on Tucano2-Instruct, then apply the validated recipe to Polygl0t/Tucano2-qwen-3.7B-Instruct.

---

9. Hyperparameter Decision Log

Parameter	Value	Rationale
model	Polygl0t/Tucano2-qwen-0.5B-Instruct	2× better benchmarks than Think; no <think> overhead; structured output proven in model card
temperature	1.0	Skywork-OR1 (2505.22312): τ=1.0 delays entropy collapse
repetition_penalty	1.0 (override from 1.2)	1.2 suppresses diversity; GRPO needs maximally diverse rollouts
num_generations	16	VRAM headroom at 0.5B allows G=16; more generations = more reward variance = stronger signal
max_completion_length	512	No <think> overhead; extraction ~100 tok, SQL ~200, insights ~300
learning_rate	2e-6	Dr. GRPO Appendix G; 4× V2's 5e-7 to push harder against APO
beta (KL)	0.0	Dr. GRPO §3.2: β=0 optimal for rule-based rewards; no ref model memory needed
scale_rewards	False	Dr. GRPO: removes std normalization bias
max_steps	200	Validation run; extend only if probe passes
lora_r	16	Standard; matches V2/V3 SFT adapter
lora_alpha	32	2× lora_r
batch_size	2	Effective batch: 2 prompts × 16 gen = 32 completions per step
grad_accum	1	Keep effective batch small for faster iteration
max_seq_length	2048	Model supports 4096; 2048 is generous for Instruct (no think overhead)
use_cache	True (override from false)	Required for O(n) autoregressive generation
top_k	0 (override from 50)	Disable top-k; let temperature alone control diversity

---

10. File Structure

tucano2_pipeline/
├── v4_instruct_grpo.ipynb             ← THE NOTEBOOK (single model, all tasks)
├── data/
│   └── pairs/
│       └── train.jsonl                ← existing full V2 training set (ALL tasks)
└── models/
    ├── tucano2-commerce-sft/          ← existing V2 SFT adapter (3.7B) — not used in V4
    └── tucano2-0.5B-instruct-grpo-v4/ ← V4 output: Instruct model GRPO adapter

No task-specific data splits needed. No Think model artifacts.

---

ADR-002 authored 2026-04-25. Based on direct audit of model repos Polygl0t/Tucano2-qwen-0.5B-Instruct and Polygl0t/Tucano2-qwen-0.5B-Think, cross-referenced with docs/INVESTIGATION_REPORT.md (20+ papers) and V1–V3 accumulated learnings.