docs/PROJECT.md

# Tucano2-Commerce: Domain-Specialized LLM for Brazilian E-Commerce Analysis

## Project Status: v2 Complete — v3 Planned

**Model:** Qwen3-3.7B → SFT → GRPO alignment  
**Domain:** Brazilian e-commerce (sentiment analysis, churn prediction, SQL generation, structured extraction)  
**Infrastructure:** Vertex AI Workbench, NVIDIA L4 (24GB), Unsloth + TRL 0.24.0  
**Tracking:** W&B project `tferrazrafael-self/tucano2-commerce`

---

## 1. Problem Statement

Brazilian e-commerce companies need automated analysis of customer reviews, churn prediction, and business intelligence generation — all in Portuguese. General-purpose LLMs (GPT-4o, Claude) are:

1. **Expensive at scale** — API costs of ~$0.01/analysis × thousands of daily reviews
2. **Not domain-optimized** — miss Brazilian Portuguese idioms, e-commerce-specific patterns
3. **Not self-hosted** — customer data leaves the organization for every API call

**Goal:** Build a compact (3.7B parameter) model that matches or exceeds large general models on e-commerce-specific tasks, runs on a single GPU, and keeps data on-premise.

---

## 2. Context & Approach

### Architecture Decision: SFT + GRPO

The training pipeline follows the DeepSeek-R1 paradigm (arxiv: 2501.12948):

```
Qwen3-3.7B (base) → SFT (domain adaptation) → GRPO (alignment via reward signals)
```

**Why Qwen3-3.7B:**
- Strong multilingual base with Portuguese capability
- 3.7B parameters fits in 24GB VRAM with 4-bit quantization (Unsloth NF4)
- Qwen3 architecture includes native `<think>` reasoning mode

**Why GRPO over DPO/PPO:**
- No need for a separate reward model (rule-based rewards suffice for structured tasks)
- Group-relative optimization naturally handles multi-task reward distributions
- Published results show GRPO working well at this model scale (Dr. GRPO, Skywork-OR1)

**Why rule-based rewards:**
- E-commerce tasks have verifiable outputs (JSON schema adherence, SQL execution, sentiment polarity)
- Neural reward models introduce reward hacking at small scale
- DeepSeek-R1 demonstrated rule-based rewards outperform neural reward models for structured tasks

### Task Portfolio

| Task | Input | Output | Evaluation |
|------|-------|--------|------------|
| Structured Extraction | Customer review + metadata | JSON with 10 fields | Field-level match |
| Sentiment Analysis | Review text | Polarity + score | Accuracy + F1 |
| SQL Generation | Business question | Executable SQL query | Execution accuracy |
| Churn Prediction | Customer profile | Risk score + reasoning | Binary accuracy |
| Business Insights | Open-ended question | Analytical report in PT-BR | LLM-as-judge |

### Infrastructure

- **Training:** Vertex AI Workbench, single NVIDIA L4 (24GB VRAM)
- **Quantization:** Unsloth NF4 (4-bit) for training — enables 3.7B model to fit in 24GB
- **Framework:** TRL 0.24.0 (pinned for Unsloth compatibility), `UnslothGRPOTrainer`
- **Monitoring:** Weights & Biases

---

## 3. Decision Log

### Decision 1: Continuous vs. Binary Reward Functions
- **Context:** Initial reward functions used binary (0/1) scoring
- **Problem:** 50% of training steps showed `reward_std=0` and `loss=0` — no learning signal
- **Decision:** Rewrote all 4 reward functions with continuous scoring (0.0–1.0), partial credit for partially correct outputs
- **Consequence:** Zero-std steps dropped from 50% to ~10%; loss became consistently non-zero
- **Reference:** Dr. GRPO paper (2503.20783) proves std-based normalization amplifies this issue

### Decision 2: Temperature 0.8 → 1.0
- **Context:** Model's `generation_config.json` had `temperature=0.1` (default from Qwen3)
- **Problem:** All 8 GRPO completions were near-identical → zero reward variance → zero advantage → zero gradient. `frac_reward_zero_std=1.0` on every step. First full run was killed.
- **Decision v2:** Set `temperature=0.8` in GRPOConfig
- **Outcome v2:** Fixed the zero-std catastrophe. Training ran 210 steps, eval improved 50% (0.083→0.125)
- **Decision v3 (planned):** Increase to `temperature=1.0` — all published GRPO papers (DeepSeek-R1, Dr. GRPO, Skywork-OR1) use 1.0. Higher temperature further delays entropy collapse.
- **Reference:** Skywork-OR1 (2505.22312) ablation: τ=1.0 gives 5-8% better test performance than τ=0.6

### Decision 3: `scale_rewards=False` (Dr. GRPO)
- **Context:** Standard GRPO normalizes advantages by `std(rewards)` per group
- **Problem:** When one group has low variance, dividing by a small std inflates its gradient contribution → training instability and bias toward "easy" prompts
- **Decision:** Disabled std normalization following Dr. GRPO paper
- **Consequence:** More stable training; combined with continuous rewards, eliminated most zero-gradient steps
- **Reference:** Dr. GRPO (2503.20783) achieved SOTA 43.3% on AIME 2024 with a 7B model using this fix

### Decision 4: Early Stopping Configuration
- **Context (v2 run 1):** `EARLY_STOPPING_PATIENCE=3`, `EVAL_STEPS=10`, `EVAL_MAX_TOKENS=256`
- **Problem:** Model killed at step 40. Eval tokens too short — model needs 500-700 tokens just for `</think>`. Eval was scoring incomplete generations. Only 30 steps of runway before early stopping fired.
- **Decision (v2 run 2):** `PATIENCE=10`, `EVAL_MAX_TOKENS=2048`, `EVAL_MAX_SAMPLES=5`
- **Consequence:** Training ran to step 210, early stopping fired correctly when eval plateaued
- **Lesson:** Early stopping parameters must account for the model's generation length requirements

### Decision 5: `MAX_STEPS` vs. `NUM_EPOCHS`
- **Context:** User set `NUM_EPOCHS=2`, `MAX_STEPS=300`
- **Clarification:** In TRL, `MAX_STEPS` takes absolute priority. With 300 prompts × 8 generations / (batch_size=4 × grad_accum=2) = 300 steps per epoch. `MAX_STEPS=300` = exactly one epoch regardless of `NUM_EPOCHS`.
- **Decision:** Keep `MAX_STEPS=300` for one clean epoch; decide on epoch 2 based on eval trajectory
- **Consequence:** Early stopping at 210 means the model trained through 70% of the data before plateauing

### Decision 6: TRL 0.24.0 Pinning
- **Context:** Unsloth requires specific TRL versions. Upgrading TRL broke vllm/torch dependencies.
- **Decision:** Pin `trl==0.24.0 --no-deps` after Unsloth installation
- **Consequence:** Stable environment, but locks out newer TRL features (e.g., entropy bonus in GRPOConfig)
- **Workaround for v3:** Implement entropy control via custom callback or trainer subclass

---

## 4. Training Results

### v2 Final Run (Step 210/300, Early Stopped)

| Metric | Value | Assessment |
|--------|-------|------------|
| `eval/best_reward_final` | 0.125 | +50% from starting 0.083 |
| `train/reward` | 0.285 | Below SFT calibration (0.38) |
| `train/frac_reward_zero_std` | 0.0 | ✅ Fixed (was 1.0 in v1) |
| `train/kl` | 0.004 | Very conservative policy shift |
| `train/clip_ratio` | 0.0 (all) | ⚠️ Entropy collapse — policy never hit clip bounds |
| `train/completion_length` | 2048 (= max) | ⚠️ Every completion truncated |
| `train/grad_norm` | 0.030 | Stable |
| Duration | 14.9 hours | 210 steps × ~4.3 min/step |

### Validation Results (5 held-out prompts)

| Sample | Task | Reward | Notes |
|--------|------|--------|-------|
| 1 | Extraction (JSON) | 0.12 | Fields incorrect, output truncated |
| 2 | Insights (categories) | 0.70 | Coherent PT-BR, structured headers |
| 3 | Retention analysis | 0.70 | Step-by-step methodology |
| 4 | Reengagement decision | 0.50 | `<think>` reasoning visible, contextual |
| 5 | Regional comparison | 0.70 | Comparative framework |

**Mean validation reward: 0.54** vs SFT calibration baseline of **0.38** → **+42% improvement**

### Bimodal Performance Pattern

- **Strong (0.50–0.70):** Open-ended analysis, insights, comparison tasks
- **Weak (0.12):** Structured JSON extraction — the completion ceiling blocks the model from outputting complete JSON

---

## 5. Diagnosed Issues & Root Causes

### Issue 1: Entropy Collapse (Critical)
- **Symptom:** `clip_ratio=0` on all steps, KL=0.004
- **Root cause:** Policy entropy drops to near-zero → all 8 rollouts produce identical output → zero advantage → zero gradient (Skywork-OR1, 2505.22312)
- **Fix (v3):** Temperature=1.0, add entropy loss coefficient (α=5e-3), filter zero-advantage groups

### Issue 2: Completion Length Ceiling (Critical)
- **Symptom:** `completion_length=2048` (= `max_completion_length`) on every step
- **Root cause:** GRPO length bias inflates incorrect response length (Dr. GRPO, 2503.20783 §3.1). Model can't finish reasoning → gets low rewards → weak signal
- **Fix (v3):** Increase `max_completion_length` to 4096, reduce `num_generations` 8→4 to fit VRAM

### Issue 3: Data Scale (Moderate)
- **Symptom:** Early stopping at step 210 (70% of epoch), eval plateaued at 0.125
- **Root cause:** 300 prompts is below published minimums (1K–600K in literature)
- **Fix (v3):** Expand to 1000+ prompts via synthetic generation and data augmentation

---

## 6. Lessons Learned

### Technical Lessons

1. **Default model configs kill RL training.** Qwen3's `generation_config.json` sets `temperature=0.1`. This single default destroyed the first full training run. Always override generation parameters explicitly.

2. **Reward function design is the core ML engineering task.** Binary rewards → zero signal. Continuous rewards → training works. Multi-component rewards (format + content + quality) → staged learning where format converges first. The reward function IS the product specification.

3. **GRPO needs diversity to learn.** The algorithm is fundamentally about comparing different completions. Anything that reduces diversity (low temperature, small group size, few prompts, completion ceiling) directly reduces learning signal.

4. **TRL step calculation is non-obvious.** `steps = num_prompts × num_generations / (batch_size × grad_accum)`. Missing the `num_generations` multiplier gives wrong epoch estimates. `MAX_STEPS` always overrides `NUM_EPOCHS`.

5. **Early stopping needs tuning for generative models.** Patience must account for eval generation length. Short eval tokens → incomplete outputs → flat eval scores → premature stopping.

6. **Entropy collapse is the GRPO failure mode.** Not divergence, not reward hacking — the model collapses to deterministic output. Monitoring `clip_ratio` and generation entropy is essential.

### Business Lessons

1. **Domain data is the moat, not model size.** ThinkJSON (1.5B) beats DeepSeek-R1 (671B) on JSON extraction. A 7B model beats o3-mini on SQL. 300 Portuguese e-commerce examples already produced a model that outperforms SFT baseline by 42%.

2. **Cost arbitrage is immediate.** A self-hosted 3.7B model on a $0.50/hr GPU costs ~$0.001/analysis vs $0.01+ for API calls. Breakeven at ~100 analyses/day.

3. **Privacy is a feature.** Self-hosted means customer data never leaves the organization. This matters for LGPD (Brazilian data protection law) compliance.

4. **Portuguese-first is defensible.** Most LLM development is English-first. A model that deeply understands Brazilian e-commerce Portuguese ("veio com defeito", "nota 1 estrela") has a real competitive advantage.

5. **Budget 3-5 iterations, not 1.** The first run is diagnostic. v1 found the zero-signal bug. v2 found the temperature bug and completion ceiling. v3 will address entropy collapse. Each iteration is cheaper than the last because you know what to measure.

---

## 7. Next Steps

See `docs/ADR-001-next-steps.md` for detailed execution plans.

### Priority 1: Build Domain Benchmark (1-2 days)
50-100 held-out prompts, automated scoring, establish baselines

### Priority 2: Run Comparison vs. Qwen3-35B-A3B (1 day)
Prove small tuned model matches/beats large general model on domain tasks

### Priority 3: GRPO v3 Training Run (2-3 days)
Fix entropy collapse, increase completion length, expand training data

---

## References

| Paper | Key Finding | Relevance |
|-------|------------|-----------|
| DeepSeek-R1 (2501.12948) | SFT→GRPO pipeline, rule-based rewards | Architecture template |
| Dr. GRPO (2503.20783) | Remove std normalization, remove length bias | Fixes reward scaling |
| Skywork-OR1 MAGIC (2505.22312) | Entropy collapse diagnosis and fix | Explains clip_ratio=0 |
| MC-GRPO (2601.22582) | Median baseline for small rollout budgets | Fixes G=8 noise |
| ThinkJSON (2502.14905) | 1.5B beats 671B on JSON extraction | Proves domain specialization thesis |
| Reasoning-SQL (2503.23157) | 7B beats o3-mini on SQL with GRPO | Proves GRPO works for SQL |
| Cocktail Effect (2410.01109) | Multi-task SFT + general data boosts domain performance | SFT improvement recipe |