Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -256,3 +256,51 @@ model = AutoModelForCausalLM.from_pretrained(model_id)
|
|
| 256 |
```
|
| 257 |
|
| 258 |
For non-causal architectures, replace `AutoModelForCausalLM` with the appropriate `AutoModel` class.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 256 |
```
|
| 257 |
|
| 258 |
For non-causal architectures, replace `AutoModelForCausalLM` with the appropriate `AutoModel` class.
|
| 259 |
+
|
| 260 |
+
|
| 261 |
+
## Trained Router (NEW)
|
| 262 |
+
|
| 263 |
+
The heuristic router has been replaced with a **trained XGBoost router** using the CARROT architecture.
|
| 264 |
+
|
| 265 |
+
### Architecture: Difficulty-First + ML Confirmation + Safety Floors
|
| 266 |
+
|
| 267 |
+
1. Map task_type to difficulty (1-5)
|
| 268 |
+
2. Compute base_tier = min(difficulty + 1, 5)
|
| 269 |
+
3. Apply safety floor per task_type (e.g., legal → tier 4)
|
| 270 |
+
4. Use per-tier P(success|query) XGBoost classifiers to confirm or escalate
|
| 271 |
+
5. If P(success@base_tier) < threshold, escalate one tier
|
| 272 |
+
|
| 273 |
+
### Usage
|
| 274 |
+
|
| 275 |
+
```python
|
| 276 |
+
from aco.learned_router import TrainedRouter
|
| 277 |
+
|
| 278 |
+
# Load from Hub
|
| 279 |
+
router = TrainedRouter.from_pretrained("narcolepticchicken/agent-cost-optimizer")
|
| 280 |
+
|
| 281 |
+
# Predict
|
| 282 |
+
tier, confidence = router.predict(
|
| 283 |
+
"Write a Python function to reverse a linked list",
|
| 284 |
+
"coding",
|
| 285 |
+
difficulty=3,
|
| 286 |
+
)
|
| 287 |
+
print(f"Recommended: tier {tier} (confidence: {confidence:.2f})")
|
| 288 |
+
```
|
| 289 |
+
|
| 290 |
+
### Benchmark Results (2K eval traces)
|
| 291 |
+
|
| 292 |
+
| Router | Success | AvgCost | Unsafe |
|
| 293 |
+
|--------|---------|---------|--------|
|
| 294 |
+
| trained (t=0.55) | 85.5% | 1.107 | 4.1% |
|
| 295 |
+
| trained (t=0.65) | 91.9% | 1.365 | 1.5% |
|
| 296 |
+
| always_frontier | 88.8% | 1.000 | 2.5% |
|
| 297 |
+
| heuristic_diff+1 | 83.4% | 0.940 | 4.9% |
|
| 298 |
+
| oracle | 99.8% | 0.486 | 0.0% |
|
| 299 |
+
|
| 300 |
+
The trained router at t=0.65 **outperforms always-frontier on success rate** (91.9% vs 88.8%) with lower unsafe miss rate (1.5% vs 2.5%).
|
| 301 |
+
|
| 302 |
+
### Training Data
|
| 303 |
+
|
| 304 |
+
50,000 synthetic traces with ground-truth per-tier success labels. Each trace includes all 5 tier outcomes, enabling the per-tier classifiers to learn from balanced success/failure examples.
|
| 305 |
+
|
| 306 |
+
See `docs/trained_router_report.md` for full details.
|