File size: 2,502 Bytes
96c57cf | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 | # BERT Router Evaluation Results
## Setup
- BERT: DistilBERTForSequenceClassification (num_labels=2, binary)
- Trained on SPROUT (31K rows, 13 models) for binary success/fail prediction
- Evaluated on SWE-Router (500 tasks, 8 models)
## Results
| Policy | Success | CostRed |
|--------|---------|---------|
| Oracle | 87.0% | 80.3% |
| v10+feedback | 81.4% | -35.6% |
| bert+feedback | 81.0% | -35.3% |
| Frontier | 78.2% | baseline |
| v10 XGBoost | 56.2% | -7.4% |
| BERT | 49.2% | -0.4% |
| Always cheap | 63.2% | 95.5% |
## Diagnosis: BERT Router is Broken
**Root cause**: BERT is a binary classifier (num_labels=2) trained to predict success/fail on SPROUT. When used for per-tier routing by prepending `[Tier X]` to the input, it ignores the tier prefix and predicts P(success) ≈ 89.5% for ALL tiers.
**Evidence**:
- All 100 sampled tasks routed to Tier 1
- P(success) is nearly identical across all tiers: 89.5% ± 0.001
- The tier prefix `[Tier X]` has no effect on BERT's predictions
**Why**: BERT was trained on SPROUT data where the input was just the problem statement, not `[Tier X] problem_statement`. The model never saw the tier prefix during training, so it ignores it.
## Fix Required
To make BERT work for tier routing, we need one of:
### Option A: Retrain as 5-class model
- Change num_labels from 2 to 5
- Labels: optimal tier (1-5) for each task
- This directly predicts the best tier from the problem statement
### Option B: Retrain with tier-prefixed inputs
- Keep binary classification
- Augment training data: for each task, create 5 examples `[Tier 1] problem`, `[Tier 2] problem`, etc.
- Label = success/fail at that tier
- This teaches the model to condition on the tier prefix
### Option C: Use BERT for feature extraction only
- Use BERT's [CLS] embedding as features for the XGBoost router
- This replaces hand-crafted keyword features with learned representations
- No need to change BERT's training
**Recommended**: Option C — it's the least risky and provides immediate value by upgrading the XGBoost feature extraction.
## v10 XGBoost Performance Issue
The v10_fixed model also underperforms expectations here (56.2% direct vs 76.6% in previous eval). This is likely because:
1. The v10_fixed model has only 14 features (fewer than the full model)
2. The threshold of 0.65 may not be well-calibrated for this model
3. The safety floor enforcement may be too weak
This needs investigation — the original v10 eval used a different model bundle.
|