docs/trained_router_final_report.md

# Trained Router Final Report

## The Honest Answer

After 7 iterations of router training (v1-v7), here is the complete picture:

### What Works

| Router | Success | AvgCost | CostRed | Unsafe |
|--------|---------|---------|---------|--------|
| **always_frontier** | 89.3% | 1.0000 | 0% | 2.3% |
| **v4_prod_t0.65** | **91.9%** | 1.3650 | -36.5% | **1.5%** |
| **heuristic_diff+1** | 84.1% | 0.9272 | 7.3% | 4.7% |
| **hybrid_v6_s0.40_d0.75** | 81.8% | 0.8222 | 17.8% | 5.9% |
| **v7_s0.25_d0.85** | 83.8% | 0.9084 | 9.2% | 4.8% |
| **oracle** | 99.8% | 0.4769 | 52.3% | 0.0% |

### What The Data Shows

1. **The heuristic (difficulty+1) is already a strong baseline** — 84.1% success at 7.3% cost reduction. The ML classifiers cannot consistently beat it because difficulty is the dominant predictive feature (12.2% importance).

2. **The ML safety net adds value in two ways:**
   - **Escalation path**: v4 at t=0.65 achieves 91.9% success — 2.6pp above frontier — by escalating when P(success) is low. This prevents unsafe cheap-model failures.
   - **Cost saving path**: v7 at d=0.85 achieves 9.2% cost reduction with only 0.3pp success loss vs heuristic.

3. **The oracle shows 52.3% cost reduction is achievable** — the gap between current routers and oracle shows there's massive room for improvement, but it requires better features than just text keywords and task type.

### Why Pure ML Doesn't Beat The Heuristic

The per-tier P(success) classifiers have:
- Tier 1: f1=0.48 (poor — success is only 22% of traces)
- Tier 2: f1=0.56 (mediocre — success is 40% of traces)
- Tier 3-5: f1=0.63-0.74 (decent — success is 70-95% of traces)

The classifiers struggle at low tiers because success at tier 1-2 is inherently rare (the model is weak). They can't reliably predict when a cheap model will succeed because the signal is weak.

### Recommendation: Use Hybrid v7_s0.25_d0.85

This configuration:
- Starts with the heuristic (difficulty + 1)
- Escalates if P(success) < 0.25 (ML safety net)
- Downgrades if P(success@tier-1) >= 0.85 (ML cost saver)

Results: **83.8% success, 9.2% cost reduction** — a meaningful improvement over the heuristic (7.3% cost reduction) with minimal quality loss (0.3pp).

### What Would Make The Router Significantly Better

1. **Execution feedback features**: Instead of predicting from text alone, use the first model call's output as a feature for subsequent routing. This is what BAAR (2026) does — profile with small model, then decide.

2. **Confidence from generation**: Get the model's own confidence (logprobs, entropy) as a routing signal. High entropy = need stronger model.

3. **Retrieval-based features**: Use retrieved similar-task traces as features. "Last time someone asked this, tier 3 failed, tier 4 succeeded."

4. **Multi-step routing**: Route per-step, not per-task. A task may start easy but get harder mid-execution.

5. **Real agent traces**: 50K synthetic traces don't capture real model behavior. Train on actual execution data from SWE-bench, BFCL, or production logs.