| # ACO Roadmap |
|
|
| ## Completed (v1-v11) |
|
|
| - [x] Normalized trace schema |
| - [x] Synthetic trace generator (10K traces) |
| - [x] Cost telemetry collector |
| - [x] Task cost classifier |
| - [x] Model cascade router (XGBoost per-tier) |
| - [x] Context budgeter |
| - [x] Cache-aware prompt layout |
| - [x] Tool-use cost gate |
| - [x] Verifier budgeter |
| - [x] Retry/recovery optimizer |
| - [x] Meta-tool miner |
| - [x] Early termination detector |
| - [x] Execution-feedback router (entropy cascade) |
| - [x] Per-step routing |
| - [x] Real benchmark evaluation (SWE-bench, BFCL) |
| - [x] Ablation study on real data |
| - [x] Literature review |
| - [x] Deployment guide |
| - [x] Technical blog post |
| - [x] Final report |
| - [x] Model cards |
|
|
| ## In Progress |
|
|
| - [ ] Fine-tuned DistilBERT router (cloud job training on SPROUT) |
| - [ ] Gradio dashboard with real benchmark numbers |
|
|
| ## Next Priority (CPU-friendly) |
|
|
| - [ ] Conformal calibration of escalation thresholds |
| - [ ] Cost-quality Pareto frontier visualization |
| - [ ] JSON schema validation for traces |
| - [ ] Unit tests for all 11 modules |
| - [ ] Integration test suite |
| - [ ] Example notebooks |
| - [ ] Provider adapter examples (OpenAI, Anthropic, local) |
| - [ ] Config file validator |
| - [ ] CLI improvements (batch routing, cost estimation) |
|
|
| ## Next Priority (GPU needed) |
|
|
| - [ ] Execution-feedback with real model logprobs |
| - [ ] Best-of-N cheap sampling with reward model |
| - [ ] Fine-tuned BERT per-step router |
| - [ ] Process reward model for selective verification |
| - [ ] Real agent benchmarks (SWE-bench Live, WebArena) |
|
|
| ## Long-term |
|
|
| - [ ] Learned context selector (vs heuristic budgeter) |
| - [ ] Workflow mining from real traces |
| - [ ] Online learning from new traces |
| - [ ] Multi-agent cost optimization |
| - [ ] Provider-aware routing (cost/latency/availability) |
| - [ ] Budget-constrained decoding |
| - [ ] Cross-task transfer learning |
|
|
| ## Known Limitations |
|
|
| - Router trained on SPROUT + SWE-Router only (need more domains) |
| - Execution feedback uses simulated logprobs (need real model outputs) |
| - No conformal guarantees on quality (hand-tuned thresholds) |
| - Per-step routing not yet integrated with v11 XGBoost |
| - Cache-aware layout not benchmarked with real providers |
| - No real agent harness integration tested end-to-end |
|
|
| ## Headroom |
|
|
| Oracle on SWE-bench shows 80.3% cost reduction is achievable. v11 achieves 36.9%. The remaining 43.4% comes from: |
| - Better per-step routing (~10%) |
| - Real execution feedback (~10%) |
| - Best-of-N cheap sampling (~8%) |
| - Conformal calibration (~5%) |
| - More training data from more domains (~10%) |
|
|
