| # BERT Router Evaluation Results |
|
|
| ## Setup |
| - BERT: DistilBERTForSequenceClassification (num_labels=2, binary) |
| - Trained on SPROUT (31K rows, 13 models) for binary success/fail prediction |
| - Evaluated on SWE-Router (500 tasks, 8 models) |
| |
| ## Results |
| |
| | Policy | Success | CostRed | |
| |--------|---------|---------| |
| | Oracle | 87.0% | 80.3% | |
| | v10+feedback | 81.4% | -35.6% | |
| | bert+feedback | 81.0% | -35.3% | |
| | Frontier | 78.2% | baseline | |
| | v10 XGBoost | 56.2% | -7.4% | |
| | BERT | 49.2% | -0.4% | |
| | Always cheap | 63.2% | 95.5% | |
| |
| ## Diagnosis: BERT Router is Broken |
| |
| **Root cause**: BERT is a binary classifier (num_labels=2) trained to predict success/fail on SPROUT. When used for per-tier routing by prepending `[Tier X]` to the input, it ignores the tier prefix and predicts P(success) ≈ 89.5% for ALL tiers. |
|
|
| **Evidence**: |
| - All 100 sampled tasks routed to Tier 1 |
| - P(success) is nearly identical across all tiers: 89.5% ± 0.001 |
| - The tier prefix `[Tier X]` has no effect on BERT's predictions |
|
|
| **Why**: BERT was trained on SPROUT data where the input was just the problem statement, not `[Tier X] problem_statement`. The model never saw the tier prefix during training, so it ignores it. |
|
|
| ## Fix Required |
|
|
| To make BERT work for tier routing, we need one of: |
|
|
| ### Option A: Retrain as 5-class model |
| - Change num_labels from 2 to 5 |
| - Labels: optimal tier (1-5) for each task |
| - This directly predicts the best tier from the problem statement |
| |
| ### Option B: Retrain with tier-prefixed inputs |
| - Keep binary classification |
| - Augment training data: for each task, create 5 examples `[Tier 1] problem`, `[Tier 2] problem`, etc. |
| - Label = success/fail at that tier |
| - This teaches the model to condition on the tier prefix |
| |
| ### Option C: Use BERT for feature extraction only |
| - Use BERT's [CLS] embedding as features for the XGBoost router |
| - This replaces hand-crafted keyword features with learned representations |
| - No need to change BERT's training |
| |
| **Recommended**: Option C — it's the least risky and provides immediate value by upgrading the XGBoost feature extraction. |
| |
| ## v10 XGBoost Performance Issue |
| |
| The v10_fixed model also underperforms expectations here (56.2% direct vs 76.6% in previous eval). This is likely because: |
| 1. The v10_fixed model has only 14 features (fewer than the full model) |
| 2. The threshold of 0.65 may not be well-calibrated for this model |
| 3. The safety floor enforcement may be too weak |
| |
| This needs investigation — the original v10 eval used a different model bundle. |
| |
