narcolepticchicken commited on
Commit
81a993a
·
verified ·
1 Parent(s): 17a2ae0

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +48 -0
README.md CHANGED
@@ -256,3 +256,51 @@ model = AutoModelForCausalLM.from_pretrained(model_id)
256
  ```
257
 
258
  For non-causal architectures, replace `AutoModelForCausalLM` with the appropriate `AutoModel` class.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
256
  ```
257
 
258
  For non-causal architectures, replace `AutoModelForCausalLM` with the appropriate `AutoModel` class.
259
+
260
+
261
+ ## Trained Router (NEW)
262
+
263
+ The heuristic router has been replaced with a **trained XGBoost router** using the CARROT architecture.
264
+
265
+ ### Architecture: Difficulty-First + ML Confirmation + Safety Floors
266
+
267
+ 1. Map task_type to difficulty (1-5)
268
+ 2. Compute base_tier = min(difficulty + 1, 5)
269
+ 3. Apply safety floor per task_type (e.g., legal → tier 4)
270
+ 4. Use per-tier P(success|query) XGBoost classifiers to confirm or escalate
271
+ 5. If P(success@base_tier) < threshold, escalate one tier
272
+
273
+ ### Usage
274
+
275
+ ```python
276
+ from aco.learned_router import TrainedRouter
277
+
278
+ # Load from Hub
279
+ router = TrainedRouter.from_pretrained("narcolepticchicken/agent-cost-optimizer")
280
+
281
+ # Predict
282
+ tier, confidence = router.predict(
283
+ "Write a Python function to reverse a linked list",
284
+ "coding",
285
+ difficulty=3,
286
+ )
287
+ print(f"Recommended: tier {tier} (confidence: {confidence:.2f})")
288
+ ```
289
+
290
+ ### Benchmark Results (2K eval traces)
291
+
292
+ | Router | Success | AvgCost | Unsafe |
293
+ |--------|---------|---------|--------|
294
+ | trained (t=0.55) | 85.5% | 1.107 | 4.1% |
295
+ | trained (t=0.65) | 91.9% | 1.365 | 1.5% |
296
+ | always_frontier | 88.8% | 1.000 | 2.5% |
297
+ | heuristic_diff+1 | 83.4% | 0.940 | 4.9% |
298
+ | oracle | 99.8% | 0.486 | 0.0% |
299
+
300
+ The trained router at t=0.65 **outperforms always-frontier on success rate** (91.9% vs 88.8%) with lower unsafe miss rate (1.5% vs 2.5%).
301
+
302
+ ### Training Data
303
+
304
+ 50,000 synthetic traces with ground-truth per-tier success labels. Each trace includes all 5 tier outcomes, enabling the per-tier classifiers to learn from balanced success/failure examples.
305
+
306
+ See `docs/trained_router_report.md` for full details.