| # ACO: Agent Cost Optimizer |
|
|
| A universal control layer that reduces autonomous agent cost while preserving task quality. |
|
|
| ## Quick Results (SWE-bench, 500 coding tasks, 8 real models) |
|
|
| | Policy | Success | Cost/Task | CostRed | |
| |--------|---------|-----------|---------| |
| | Oracle | 87.0% | $0.062 | 80.3% | |
| | **v10+feedback** | **84.8%** | **$0.201** | **36.4%** | |
| | v10 direct | 76.6% | $0.188 | 40.7% | |
| | Always frontier | 78.2% | $0.317 | baseline | |
| | Always cheap | 63.2% | $0.014 | 95.5% | |
|
|
| **Key finding: v10+feedback strictly dominates always-frontier** — lower cost AND higher quality. This is not a cost-quality tradeoff. |
|
|
| ## BERT Router Results |
|
|
| DistilBERT was fine-tuned on SPROUT for binary classification. The binary classifier fails for tier routing — it ignores tier prefixes and predicts P(success) ≈ 89.5% for all tiers, routing everything to the cheapest model. |
|
|
| A 5-class retraining is in progress (job `69fd8cccaff1cd33e8f30714`). |
|
|
| ## 11 Modules |
|
|
| 1. Cost Telemetry Collector — `aco/telemetry.py` |
| 2. Task Cost Classifier — `aco/classifier.py` |
| 3. Model Cascade Router (XGBoost + isotonic) — `aco/router_v10.py` |
| 4. Execution-Feedback Router (entropy cascade) — `aco/execution_feedback.py` |
| 5. Context Budgeter — `aco/context_budgeter.py` |
| 6. Cache-Aware Prompt Layout — `aco/cache_layout.py` |
| 7. Tool-Use Cost Gate — `aco/tool_gate.py` |
| 8. Verifier Budgeter — `aco/verifier_budgeter.py` |
| 9. Retry/Recovery Optimizer — `aco/retry_optimizer.py` |
| 10. Meta-Tool Miner — `aco/meta_tool_miner.py` |
| 11. Doom Detector — `aco/doom_detector.py` |
|
|
| ## New Modules (this session) |
|
|
| - **Conformal Calibration** — `aco/conformal.py` — RouteNLP-style distribution-free escalation guarantees |
| - **Pareto Frontier** — `aco/pareto.py` — RouterBench NDCH + RouteLLM CPT/APGR metrics |
| - **Integration Test** — `tests/test_integration.py` — Full pipeline test |
|
|
| ## Key Takeaway |
|
|
| Training on real execution data matters more than architecture. v8 trained on synthetic data *increased* cost by 11.6%. v10 trained on 500 real SWE-Router outcomes *saved* 36.4%. Same XGBoost, same features. |
|
|
| ## Documentation |
|
|
| - [Final Report](docs/final_report_v2.md) |
| - [Pareto Frontier Report](docs/pareto_frontier_report.md) |
| - [Conformal Calibration Report](docs/conformal_report.md) |
| - [BERT Eval Report](docs/bert_eval_report.md) |
| - [Literature Review](docs/literature_review.md) |
| - [Deployment Guide](docs/deployment_guide.md) |
| - [Technical Blog](docs/technical_blog.md) |
| - [Roadmap](docs/ROADMAP.md) |
|
|
| ## Links |
|
|
| - **Model**: [narcolepticchicken/agent-cost-optimizer](https://huggingface.co/narcolepticchicken/agent-cost-optimizer) |
| - **Dataset**: [narcolepticchicken/agent-cost-traces](https://huggingface.co/datasets/narcolepticchicken/agent-cost-traces) |
| - **Dashboard**: [narcolepticchicken/aco-dashboard](https://huggingface.co/spaces/narcolepticchicken/aco-dashboard) |
|
|
