File size: 21,304 Bytes

b47b36b

# ADR-001: Tucano2-Commerce Next Steps — Detailed Execution Plans

**Status:** Accepted  
**Date:** 2025-04-23  
**Author:** Rafael Ferraz  
**Context:** GRPO v2 training completed (210/300 steps, early stopped). Mean validation reward 0.54 (+42% vs SFT baseline). Three critical issues diagnosed: entropy collapse, completion length ceiling, data scale. This ADR details the execution plan for the next phase.

---

## Phase 1: Build the Domain Benchmark

**Goal:** Create a rigorous, reproducible evaluation suite that measures Tucano2 across all 5 task types.  
**Time estimate:** 1-2 days  
**Prerequisite:** None — can start immediately

### Step 1.1: Design the Benchmark Prompts

Create **80 held-out prompts** (never seen in training), stratified by task:

| Task | Count | Source | Difficulty Mix |
|------|-------|--------|---------------|
| Structured Extraction | 20 | Real reviews from Olist/B2W datasets | 10 easy (clear sentiment) + 10 hard (mixed/ambiguous) |
| Sentiment Analysis | 15 | Real reviews, balanced pos/neg/neutral | 5 per polarity |
| SQL Generation | 15 | Business questions against your e-commerce schema | 5 simple SELECT + 5 JOIN + 5 aggregate/window |
| Churn/Risk Prediction | 15 | Customer profiles with known outcomes | 5 low-risk + 5 medium + 5 high-risk |
| Business Insights | 15 | Open-ended analytical questions | 5 comparison + 5 trend + 5 recommendation |

**Implementation:**

```python
# File: benchmark/prompts.jsonl
# Each line is a JSON object:
{
    "id": "ext-001",
    "task": "extraction",
    "difficulty": "hard",
    "prompt": "Analise esta avaliação...",
    "system": "<your system prompt>",
    "ground_truth": {
        "sentiment": "negativo",
        "sentiment_score": -0.6,
        "churn_risk": 0.8,
        ...
    },
    "notes": "Mixed sentiment — product good but delivery terrible"
}
```

**Key principles:**
- Include edge cases: mixed sentiment, sarcasm, regional slang ("barato saiu caro"), incomplete reviews
- SQL prompts must have executable ground truth queries against your actual schema
- Insights prompts should require multi-step reasoning, not one-hop lookups

### Step 1.2: Build Automated Scoring Functions

Each task type gets its own scorer:

#### Extraction Scorer
```python
def score_extraction(prediction: dict, ground_truth: dict) -> dict:
    """Score each of the 10 JSON fields independently."""
    scores = {}
    
    # Categorical fields: exact match
    for field in ["sentiment", "complaint_category"]:
        scores[field] = 1.0 if pred[field] == gt[field] else 0.0
    
    # Numeric fields: distance-based
    for field in ["sentiment_score", "churn_risk", "repeat_intent"]:
        scores[field] = max(0, 1.0 - abs(pred[field] - gt[field]))
    
    # Boolean fields: exact match
    for field in ["delivery_issue", "product_issue", "seller_issue", "would_recommend"]:
        scores[field] = 1.0 if pred[field] == gt[field] else 0.0
    
    # Text fields: semantic similarity (sentence-transformers)
    for field in ["main_complaint"]:
        scores[field] = cosine_similarity(embed(pred[field]), embed(gt[field]))
    
    scores["mean"] = sum(scores.values()) / len(scores)
    return scores
```

#### SQL Scorer
```python
def score_sql(predicted_sql: str, ground_truth_sql: str, db_connection) -> dict:
    """Execute both queries and compare result sets."""
    try:
        pred_results = db_connection.execute(predicted_sql).fetchall()
        gt_results = db_connection.execute(ground_truth_sql).fetchall()
        
        # Execution accuracy (EX): do the result sets match?
        ex = 1.0 if set(pred_results) == set(gt_results) else 0.0
        
        # Partial credit: row overlap
        overlap = len(set(pred_results) & set(gt_results))
        partial = overlap / max(len(gt_results), 1)
        
        return {"ex": ex, "partial": partial, "syntax_valid": 1.0}
    except Exception:
        return {"ex": 0.0, "partial": 0.0, "syntax_valid": 0.0}
```

#### Sentiment Scorer
```python
def score_sentiment(prediction: str, ground_truth: str) -> dict:
    """Exact match on polarity, distance on score."""
    polarity_match = 1.0 if pred_polarity == gt_polarity else 0.0
    score_distance = max(0, 1.0 - abs(pred_score - gt_score) / 2.0)
    return {"polarity": polarity_match, "score": score_distance}
```

#### Churn Scorer
```python
def score_churn(prediction: float, ground_truth: float, threshold=0.5) -> dict:
    """Binary accuracy + calibration."""
    binary = 1.0 if (prediction >= threshold) == (ground_truth >= threshold) else 0.0
    calibration = max(0, 1.0 - abs(prediction - ground_truth))
    return {"binary_accuracy": binary, "calibration": calibration}
```

#### Insights Scorer (LLM-as-Judge)
```python
JUDGE_PROMPT = """You are evaluating a Brazilian e-commerce analysis response.
Rate on 4 dimensions (1-5 each):
1. Relevance: Does it address the question?
2. Accuracy: Are the claims factually reasonable?
3. Actionability: Could a business act on this analysis?
4. Portuguese Quality: Is the PT-BR natural and professional?

Response to evaluate:
{response}

Question asked:
{question}

Return JSON: {"relevance": X, "accuracy": X, "actionability": X, "portuguese": X}"""

def score_insights(response: str, question: str) -> dict:
    """Use GPT-4o as judge, run 3x and average."""
    scores = []
    for _ in range(3):
        result = openai.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": JUDGE_PROMPT.format(
                response=response, question=question
            )}],
            temperature=0.3
        )
        scores.append(json.loads(result.choices[0].message.content))
    
    # Average across 3 runs
    return {k: sum(s[k] for s in scores) / 3 for k in scores[0]}
```

### Step 1.3: Run the Benchmark Script

```python
# File: benchmark/run_benchmark.py
import json
from pathlib import Path

def run_benchmark(model, tokenizer, prompts_path, output_path):
    prompts = [json.loads(line) for line in open(prompts_path)]
    results = []
    
    for prompt in prompts:
        # Generate response
        messages = [
            {"role": "system", "content": prompt["system"]},
            {"role": "user", "content": prompt["prompt"]}
        ]
        response = generate(model, tokenizer, messages, max_new_tokens=2048, temperature=0.1)
        
        # Score based on task type
        scorer = SCORERS[prompt["task"]]
        score = scorer(parse_response(response), prompt.get("ground_truth"))
        
        results.append({
            "id": prompt["id"],
            "task": prompt["task"],
            "difficulty": prompt["difficulty"],
            "response": response,
            "scores": score,
            "tokens": len(tokenizer.encode(response))
        })
    
    # Save results
    with open(output_path, "w") as f:
        json.dump(results, f, indent=2, ensure_ascii=False)
    
    # Print summary
    for task in ["extraction", "sentiment", "sql", "churn", "insights"]:
        task_results = [r for r in results if r["task"] == task]
        mean_score = sum(r["scores"]["mean"] for r in task_results) / len(task_results)
        print(f"{task}: {mean_score:.3f} (n={len(task_results)})")

SCORERS = {
    "extraction": score_extraction,
    "sentiment": score_sentiment,
    "sql": score_sql,
    "churn": score_churn,
    "insights": score_insights,
}
```

### Step 1.4: Establish Baselines

Run the benchmark on:
1. **Qwen3-3.7B base** (zero-shot, no adapters) — this is your floor
2. **Tucano2-SFT** (SFT adapter only) — this is your pre-GRPO baseline
3. **Tucano2-GRPO v2** (current best) — this is what you're improving

Record all results in a comparison table.

---

## Phase 2: Run Comparison vs. Qwen3-35B-A3B

**Goal:** Prove domain-tuned 3.7B matches or beats general 35B on e-commerce tasks.  
**Time estimate:** 1 day (after benchmark is built)  
**Prerequisite:** Phase 1 complete

### Step 2.1: Set Up Qwen3-35B-A3B

This is a Mixture-of-Experts model (35B total, ~3B active per token). It should fit on your L4 with 4-bit quantization.

```python
from unsloth import FastLanguageModel

model_35b, tokenizer_35b = FastLanguageModel.from_pretrained(
    model_name="Qwen/Qwen3-35B-A3B",  # Verify exact HF repo name
    max_seq_length=4096,
    load_in_4bit=True,
    dtype=None,  # Auto-detect
)
FastLanguageModel.for_inference(model_35b)
```

**Memory estimate:** 35B params × 0.5 bytes (4-bit) ≈ 17.5GB. Active inference ~3B ≈ 3GB compute. Should fit on L4 (24GB) with headroom.

**If it doesn't fit:** Use `transformers` with `BitsAndBytesConfig(load_in_4bit=True)` instead of Unsloth, or try GGUF via `llama.cpp`.

### Step 2.2: Run the Same Benchmark

```python
# Run identical benchmark on Qwen3-35B-A3B
run_benchmark(model_35b, tokenizer_35b, "benchmark/prompts.jsonl", "results/qwen3-35b.json")
```

**Critical:** Use the same system prompt, same temperature (0.1 for eval), same max_new_tokens. The only variable should be the model.

### Step 2.3: Optional — Add GPT-4o Baseline

```python
# For the strongest reference point
import openai

def run_benchmark_api(prompts_path, output_path, model="gpt-4o"):
    prompts = [json.loads(line) for line in open(prompts_path)]
    results = []
    
    for prompt in prompts:
        response = openai.chat.completions.create(
            model=model,
            messages=[
                {"role": "system", "content": prompt["system"]},
                {"role": "user", "content": prompt["prompt"]}
            ],
            temperature=0.1
        )
        # ... score same as above
```

### Step 2.4: Build the Comparison Report

```
┌──────────────┬────────────┬────────────┬────────────┬────────────┬──────────┐
│ Task         │ Qwen3-3.7B │ Tucano2-SFT│ Tucano2-   │ Qwen3-35B  │ GPT-4o   │
│              │ (base)     │            │ GRPO v2    │ (zero-shot)│(zero-shot│
├──────────────┼────────────┼────────────┼────────────┼────────────┼──────────┤
│ Extraction   │            │            │            │            │          │
│ Sentiment    │            │            │            │            │          │
│ SQL          │            │            │            │            │          │
│ Churn        │            │            │            │            │          │
│ Insights     │            │            │            │            │          │
├──────────────┼────────────┼────────────┼────────────┼────────────┼──────────┤
│ MEAN         │            │            │            │            │          │
├──────────────┼────────────┼────────────┼────────────┼────────────┼──────────┤
│ Cost/query   │ $0.001     │ $0.001     │ $0.001     │ $0.003     │ $0.010   │
│ Latency (s)  │            │            │            │            │          │
│ Privacy      │ ✅ On-prem │ ✅ On-prem │ ✅ On-prem │ ✅ On-prem │ ❌ API   │
└──────────────┴────────────┴────────────┴────────────┴────────────┴──────────┘
```

**The story to tell:**
- If Tucano2-GRPO ≥ Qwen3-35B on extraction/sentiment/SQL: "Domain tuning eliminates the need for 10× larger models"
- If Tucano2-GRPO < Qwen3-35B on insights: "Open-ended reasoning still benefits from scale, but structured tasks don't"
- Cost column proves the business case regardless of performance parity

---

## Phase 3: GRPO v3 Training Run

**Goal:** Fix entropy collapse, break through the v2 performance plateau.  
**Time estimate:** 2-3 days (including data prep)  
**Prerequisite:** Phase 1 (benchmark needed to measure improvement)

### Step 3.1: Expand Training Data (1000+ prompts)

**Target: 1000 prompts** (up from 300), stratified:

| Task | Current | Target | How to Expand |
|------|---------|--------|--------------|
| Extraction | ~75 | 300 | Generate from Olist dataset — sample reviews, create ground truth JSON |
| Sentiment | ~60 | 200 | Sample from B2W reviews corpus, label with existing model + human review |
| SQL | ~75 | 250 | Template-based: vary table names, WHERE clauses, aggregation patterns |
| Churn | ~45 | 100 | Augment customer profiles with synthetic variations |
| Insights | ~45 | 150 | LLM-generated analytical questions about e-commerce scenarios |

**Synthetic generation recipe (from Cocktail Effect paper):**
```python
# Use your SFT model or GPT-4o to generate new training prompts
# Then manually verify/correct the ground truth
def generate_extraction_prompts(reviews_df, n=200):
    prompts = []
    for _, row in reviews_df.sample(n).iterrows():
        prompt = format_extraction_prompt(row["review_text"], row["score"], row["status"])
        # Generate ground truth with GPT-4o (higher quality than self-labeling)
        ground_truth = gpt4o_extract(prompt)
        prompts.append({"prompt": prompt, "ground_truth": ground_truth})
    return prompts
```

**Also add 30% general reasoning data** (Cocktail Effect paper finding):
- Source: Portuguese subset of Orca-Math or translated OpenOrca
- Purpose: Regularization — prevents model from overfitting to domain patterns
- Mix ratio: 700 domain + 300 general = 1000 total

### Step 3.2: Configuration Changes for v3

```python
# === GRPO v3 Config Changes ===

# FIX 1: Temperature — prevent entropy collapse
TEMPERATURE = 1.0  # Was 0.8 in v2. All published GRPO papers use 1.0.
# Reference: Skywork-OR1 (2505.22312) ablation shows τ=1.0 >> τ=0.6

# FIX 2: Completion length — remove the ceiling
MAX_COMPLETION_LENGTH = 4096  # Was 2048. Every v2 completion hit the ceiling.
# Trade-off: halve num_generations to fit VRAM

# FIX 3: Reduce generations to fit longer completions
NUM_GENERATIONS = 4  # Was 8. 4 × 4096 ≈ 8 × 2048 in VRAM terms.
# MC-GRPO paper shows G=4 can work if using median baseline

# FIX 4: Explicit β=0 (no KL penalty)
# Dr. GRPO paper: β=0 is optimal for rule-based rewards
BETA = 0.0

# FIX 5: Learning rate — slightly more aggressive
LEARNING_RATE = 3e-6  # Was 2e-6. Clip ratios were all 0 → room to push harder.
# Still well within published range (1e-6 to 5e-6)

# FIX 6: Max steps for expanded data
# 1000 prompts × 4 generations / (4 batch × 2 accum) = 500 steps/epoch
MAX_STEPS = 500

# FIX 7: Early stopping — more generous
EARLY_STOPPING_PATIENCE = 15  # 15 evals × 10 steps = 150 steps of runway
EVAL_STEPS = 10
SAVE_STEPS = 10
```

### Step 3.3: Add Entropy Monitoring (Critical for v3)

Since TRL 0.24.0 doesn't have native entropy bonus, implement via callback:

```python
class EntropyMonitorCallback(TrainerCallback):
    """Monitor policy entropy to detect collapse early."""
    
    def on_log(self, args, state, control, logs=None, **kwargs):
        if logs and "train/completion_length" in logs:
            # Proxy for entropy: if all completions are max length,
            # entropy is collapsing (model is stuck in one mode)
            completion_ratio = logs["train/completion_length"] / MAX_COMPLETION_LENGTH
            
            if completion_ratio > 0.95:
                print(f"⚠️ Step {state.global_step}: Completion ratio {completion_ratio:.2f} — "
                      f"possible entropy collapse. Monitor reward_std.")
            
            # Log to W&B
            if wandb.run:
                wandb.log({
                    "monitor/completion_ratio": completion_ratio,
                    "monitor/entropy_proxy": 1.0 - completion_ratio,
                }, step=state.global_step)
```

### Step 3.4: Add Zero-Advantage Group Filtering

```python
# In the reward function wrapper or custom trainer:
def filtered_commerce_reward_fn(completions, prompts, **kwargs):
    """Compute rewards and flag zero-variance groups for filtering."""
    rewards = commerce_reward_fn(completions, prompts, **kwargs)
    
    # Group rewards by prompt (each prompt has NUM_GENERATIONS completions)
    for i in range(0, len(rewards), NUM_GENERATIONS):
        group = rewards[i:i+NUM_GENERATIONS]
        if max(group) - min(group) < 0.01:  # Near-zero variance
            # Add small noise to break the tie
            # This prevents the GRPO denominator from exploding
            for j in range(i, i+NUM_GENERATIONS):
                rewards[j] += random.gauss(0, 0.005)
    
    return rewards
```

**Note:** The proper fix is MC-GRPO's median baseline, but that requires modifying TRL internals. The noise injection is a pragmatic workaround for TRL 0.24.0.

### Step 3.5: Reward Function Refinement

Split the composite reward into staged components (Reasoning-SQL paper finding):

```python
def commerce_reward_fn_v3(completions, prompts, **kwargs):
    """Multi-component reward with staged convergence."""
    rewards = []
    for completion, prompt in zip(completions, prompts):
        # Stage 1: Format reward (converges first)
        r_format = score_format(completion)  # JSON valid? Think tags closed? Right structure?
        
        # Stage 2: Partial content reward
        r_partial = score_partial_content(completion, prompt)  # Some fields correct?
        
        # Stage 3: Full task reward  
        r_task = score_full_task(completion, prompt)  # All fields correct? SQL executes?
        
        # Weighted combination — format weight decreases over training
        reward = 0.2 * r_format + 0.3 * r_partial + 0.5 * r_task
        rewards.append(reward)
    
    return rewards
```

### Step 3.6: VRAM Budget for v3

```
L4 GPU: 24GB total

Model (NF4):          ~3.5GB
KV Cache (4096 seq):  ~2.0GB
Activations:          ~4.0GB
Optimizer states:     ~3.0GB
Generations (4×4096): ~8.0GB
─────────────────────────────
Estimated total:      ~20.5GB
Headroom:             ~3.5GB

✅ Should fit. If OOM: reduce MAX_COMPLETION_LENGTH to 3072 first.
```

### Step 3.7: Training Execution

```bash
# Pre-flight checklist:
# ✅ Benchmark built and baselines recorded (Phase 1)
# ✅ 1000+ prompts prepared and validated
# ✅ Config changes applied (temperature, completion length, generations, LR)
# ✅ Entropy monitor callback added
# ✅ Zero-advantage filtering active
# ✅ Reward function v3 with staged components
# ✅ FRESH=True (nuke old checkpoints)
# ✅ W&B run name: grpo-v3-l4-{timestamp}

# Expected runtime: 500 steps × ~5 min/step (longer completions) ≈ 42 hours
# Checkpoints every 10 steps
# Early stopping patience: 15 evals (150 steps)
```

### Step 3.8: Post-Training Validation

1. Run Phase 1 benchmark on GRPO v3 best checkpoint
2. Compare against all baselines:
   - Qwen3-3.7B base → Tucano2-SFT → Tucano2-GRPO-v2 → **Tucano2-GRPO-v3**
3. If v3 > v2 on benchmark: save as production model
4. If v3 ≈ v2: entropy collapse not fully fixed; consider switching to MC-GRPO or upgrading GPU for longer completions
5. If v3 < v2: rollback, investigate — likely reward function regression

---

## Decision Criteria for Stopping

| Condition | Action |
|-----------|--------|
| v3 eval reward > 0.20 AND extraction score > 0.40 | Ship it — significantly better than v2 |
| v3 eval reward 0.15-0.20, improving trend | Run epoch 2 (extend MAX_STEPS to 1000) |
| v3 eval reward < v2 (0.125) | Stop, diagnose, review reward function and data |
| Entropy collapse again (clip_ratio=0 after step 50) | Add entropy bonus via custom loss (requires TRL fork) |
| OOM | Reduce MAX_COMPLETION_LENGTH to 3072 → 2560 → 2048 |

---

## Appendix: Literature References for Each Fix

| Fix | Paper | Section | Key Finding |
|-----|-------|---------|-------------|
| Temperature=1.0 | Skywork-OR1 (2505.22312) | §4, Table 3 | τ=1.0 gives 5-8% better performance, delays entropy collapse |
| β=0 (no KL) | Dr. GRPO (2503.20783) | §3.2 | KL penalty unnecessary for rule-based rewards |
| scale_rewards=False | Dr. GRPO (2503.20783) | §3.1 | Std normalization biases toward low-variance groups |
| Longer completions | Dr. GRPO (2503.20783) | §3.1 | GRPO length bias inflates wrong answers → ceiling hit |
| Zero-advantage filtering | Skywork-OR1 (2505.22312) | §3.1 | Zero-std groups destabilize training |
| Staged rewards | Reasoning-SQL (2503.23157) | §3.2 | Format rewards converge first, enable task learning |
| General data mixing | Cocktail Effect (2410.01109) | §4 | 30% general data improves domain performance 2-15% |
| G=4 with median | MC-GRPO (2601.22582) | §3 | Median baseline reduces noise at small group sizes |