Upload docs/final_report.md

docs/final_report.md

@@ -1,273 +1,126 @@
-# Agent Cost Optimizer — Final Technical Report
+# ACO Final Report: Agent Cost Optimizer
 
-**Date:** 2025-07-05  
-**Authors:** ML Intern (autonomous researcher)  
-**Repository:** https://huggingface.co/narcolepticchicken/agent-cost-optimizer  
-**Benchmark:** N=2,000 synthetic traces across 19 scenarios  
-**Baseline Comparison:** always_frontier, always_cheap, static, cascade, full_optimizer + 5 ablations
-
----
+## Executive Summary
 
-## 1. Executive Summary
+ACO is a universal control layer that reduces autonomous agent cost while preserving task quality. After 11 iterations of router design, training on synthetic data, real execution data, and combined datasets, the key finding is:
 
-The Agent Cost Optimizer (ACO) is a **deployable control layer** that reduces autonomous agent run costs by 28% while maintaining the same task success rate (94.3%) as a frontier-only baseline. It is not a model — it is a **compound decision system** comprising 10 interlocking modules that make cost-aware decisions at every step of an agent run.
+**Training on real execution data is the single most important lever.** The router trained on synthetic data actually *increased* cost by 11.6% on real tasks. The router trained on real SWE-Router data achieved 36.9% cost reduction at comparable quality — a 34.9 percentage point swing from one change.
 
-### Top-Line Result
-- **Cost per successful task: $0.2089** (full optimizer) vs $0.2907 (always frontier)
-- **Total cost on 2,000 tasks: $393.98** vs $548.31 (28.1% reduction)
-- **Success rate: 94.3%** (identical to frontier baseline)
-- **No regressions in quality** — false-DONE rate, unsafe cheap-model miss rate, and missed escalation rate are all preserved or improved
+## Required Answers
 
----
-
-## 2. Core Architecture
-
-ACO consists of 10 modules sharing a single normalized trace schema:
-
-| # | Module | Function | Key Decision |
-|---|--------|----------|--------------|
-| 1 | Cost Telemetry Collector | Structured trace recording | What happened, what it cost |
-| 2 | Task Cost Classifier | Task risk/cost prediction | What tier is needed |
-| 3 | Model Cascade Router | Dynamic model selection | Which model to use |
-| 4 | Context Budgeter | Context selection | What to include/exclude/summarize |
-| 5 | Cache-Aware Prompt Layout | Prefix/suffix optimization | How to structure for cache reuse |
-| 6 | Tool-Use Cost Gate | Tool worthiness prediction | Whether to call, skip, batch |
-| 7 | Verifier Budgeter | Selective verification | When to verify |
-| 8 | Retry/Recovery Optimizer | Intelligent failure recovery | How to recover |
-| 9 | Meta-Tool Miner | Workflow compression | Whether to use cached workflow |
-| 10 | Doom Detector | Early failure detection | Whether to stop |
-
-### Integration Patterns
-
-Three bolt-on patterns are supported:
-
-- **Front Proxy**: `optimize()` before each agent step
-- **Around Wrapper**: `optimize()` pre-step + `record_step()` post-step
-- **Inside Agent**: Per-step `reassess()` for mid-run adjustment
-
----
-
-## 3. Benchmark Design
-
-### 19 Realistic Scenarios
-
-The synthetic benchmark spans all major cost-waste patterns identified in the literature:
-
-| Scenario | Frequency | Pattern |
-|----------|-----------|---------|
-| quick_answer_success | 18% | Cheap model is sufficient |
-| coding_success_medium | 10% | Medium model succeeds |
-| coding_success_frontier | 8% | Frontier required |
-| coding_cheap_fail | 5% | Cheap model fails, should escalate |
-| coding_tool_underuse | 4% | Tool not called when needed |
-| research_success | 10% | Research tasks at medium tier |
-| research_cheap_fail | 3% | Research too hard for cheap model |
-| legal_frontier_success | 4% | High-risk, frontier required |
-| legal_cheap_unsafe | 2% | Unsafe cheap model on legal |
-| tool_heavy_success | 6% | Tools used efficiently |
-| retrieval_success | 6% | Retrieval-heavy tasks |
-| long_horizon_success | 5% | Multi-step tasks |
-| long_horizon_retry_loop | 3% | Retry loops wasting cost |
-| unknown_ambiguous_success | 3% | Ambiguous tasks resolved |
-| unknown_ambiguous_blocked | 2% | Should escalate to human |
-| tool_overuse | 4% | Unnecessary tool calls |
-| cache_break_scenario | 3% | Cache-unfriendly layouts |
-| false_done_scenario | 2% | Agent says done but isn't |
-| quick_answer_cheap_fail | 2% | Even quick answers can fail |
-
-### Simulation Model
-
-Success probability is modeled as `strength^difficulty`, where:
-- **Strength**: 0.35 (tiny), 0.55 (cheap), 0.80 (medium), 0.93 (frontier), 0.97 (specialist)
-- **Difficulty**: 1 (quick answer) to 5 (legal/safety-critical)
-
-This exponential relationship captures the real-world phenomenon that harder tasks need exponentially stronger models.
-
----
-
-## 4. Results
-
-### Baseline Comparison (N=2,000)
-
-| Baseline | Success | Cost/Success | Total Cost | Cost Reduction | False-DONE |
-|----------|---------|--------------|-----------|---------------|------------|
-| always_frontier | 94.3% | $0.2907 | $548.31 | 0% | 1.9% |
-| always_cheap | 16.2% | $0.2531 | $82.25 | 85.0% | 1.9% |
-| static | 73.6% | $0.2462 | $362.43 | 33.9% | 1.9% |
-| cascade | 73.9% | $0.2984 | $440.98 | 19.6% | 1.9% |
-| **full_optimizer** | **94.3%** | **$0.2089** | **$393.98** | **28.1%** | **1.9%** |
-
-### Ablation Study
-
-| Module Removed | Success | Cost/Success | Cost Increase | Impact |
-|---------------|---------|--------------|---------------|--------|
-| no_router | 73.6% | $0.2462 | -8.7% | **+20.7pp quality loss** |
-| no_tool_gate | 69.8% | $0.2596 | -8.0% | **+24.5pp quality loss** |
-| no_verifier | 71.1% | $0.2549 | -8.0% | **+23.2pp quality loss** |
-| no_early_term | 73.6% | $0.2488 | -6.8% | +20.7pp quality loss |
-| no_context_budget | 73.6% | $0.2462 | -8.7% | +20.7pp quality loss |
-
-**Key Finding:** The ablations show that removing *any* single module drops success rate to ~70–74%. This is not because each module individually saves money — it's because the modules interact. The router avoids putting hard tasks on cheap models; the verifier catches mistakes that would have been caught by expensive models; the tool gate prevents wasted tool calls that the router already optimized for. The full_optimizer is greater than the sum of its parts.
-
-### Quality/Cost Frontier
-
-Pareto-optimal configurations:
-
-1. **full_optimizer**: 94.3% success at $0.2089/success ← **Best overall**
-2. **always_frontier**: 94.3% success at $0.2907/success ← Maximum quality, 28% more expensive
-3. **static**: 73.6% success at $0.2462/success ← Budget option
-
-`always_cheap` and `cascade` are **not Pareto-optimal** — they are dominated by `full_optimizer` (better quality at lower or equal cost).
-
----
+### How much cost was saved at iso-quality?
 
-## 5. Answering the Required Questions
+On real SWE-bench tasks (500 coding tasks, 8 models):
 
-### How much cost was saved at iso-quality?
+| Comparison | Cost Reduction | Quality Delta |
+|-----------|---------------|---------------|
+| v11 feedback vs always-frontier | 36.9% | -3.4pp (74.8% vs 78.2%) |
+| v11 cascade (thr=0.65) vs frontier | 62.5% | -10.8pp (67.4% vs 78.2%) |
+| v9 feedback (synthetic) vs frontier | 2.1% | +0.1pp (90.0% vs 90.0%) |
 
-**28.1% reduction** ($0.2907 → $0.2089 per successful task) with identical 94.3% success rate vs always-frontier baseline. Total savings: $154.33 on 2,000 tasks.
+On synthetic benchmarks, v9 with execution feedback matches frontier quality exactly (90.0% vs 90.0%) at 2.1% cost reduction. On real data, the quality gap is wider because real agent tasks have longer horizons and more failure modes.
 
 ### Which module saved the most?
 
-The **Model Cascade Router** has the highest individual impact. When removed (`no_router`), success rate drops by 20.7 percentage points while cost-per-success *decreases* slightly — this means the router is the primary quality-preserver. Without it, the system saves a few cents but fails catastrophically on hard tasks.
+**Ablation results (SWE-bench, 500 tasks):**
 
-However, **no module is dominant in isolation** — the ablations show that removing *any* module causes large quality regressions. The system is designed as a compound optimizer where modules reinforce each other.
+| Module Removed | Success Delta | Cost Delta | Impact |
+|---------------|---------------|------------|--------|
+| Feedback escalation | -8.6pp | -$0.0027 | Highest quality impact |
+| v10/v11 router (vs heuristic) | +14.8pp | -$0.024 | Highest cost impact |
+| Execution-feedback (v9) | +6.2pp | +$0.063 | Matches frontier quality |
+
+The **model cascade router** saves the most cost. The **execution-feedback escalation** preserves the most quality. They are synergistic — removing either one causes significant regression.
 
 ### Which module caused regressions?
 
-No module caused regressions in the full_optimizer configuration. The ablations show regressions only when modules are *removed*. All 10 modules contribute positively to the cost-quality frontier.
+- **Aggressive v3 asymmetric router**: Over-penalized underkill, causing over-escalation and 38% cost increase
+- **v8 synthetic-trained router**: On real data, it actually increased cost by 11.6% because synthetic success probabilities don't match real execution outcomes
+- **Over-aggressive feedback (v9 entropy_thr=2.0)**: Escalates too often, paying for both cheap and frontier models, resulting in -53% cost reduction (costs more than frontier alone)
 
 ### When should the optimizer use cheap models?
 
-Per the Task Cost Classifier and Router rules:
-- **Quick answers** (difficulty 1): tier 2 (GPT-4o-mini, Claude-Haiku)
-- **Document drafting** (difficulty 2): tier 2–3
-- **Coding, research, long-horizon** (difficulty 3–4): tier 3 (Claude-3.5-Sonnet, DeepSeek)
-- **Legal, regulated, safety-critical** (difficulty 5): tier 4–5 only
-- **When confidence is high** (prior success rate >80% on similar tasks)
-- **When the task is reversible** (no irreversible actions planned)
+- Quick answers, simple lookups, arithmetic
+- Tasks with "typo", "simple", "minor", "just" keywords
+- Search and read steps in multi-step agent runs
+- Late steps in a run (context is already built up)
+- 67.4% of SWE-bench tasks succeed at tier 1
 
 ### When should it force frontier models?
 
-- **Legal/regulated tasks** (difficulty 5, risk >0.7)
-- **Irreversible actions** (deploy, delete, financial transactions)
-- **Low confidence** (<0.6) on ambiguous tasks
-- **Prior failures** on similar tasks
-- **Verifier disagreement** (backstop when cheap model was used)
-- **Safety-critical** (medical, financial, legal)
+- Legal/regulated tasks (safety floor = tier 4)
+- Critical production issues ("production", "urgent", "emergency" keywords)
+- Edit/patch steps on security-critical code
+- Verification of high-risk outputs
+- When cheap model already failed (escalation)
 
 ### When should it call a verifier?
 
-Per the Verifier Budgeter:
-- **High-risk tasks** (legal, compliance, safety)
-- **Low confidence** in model output (<0.7)
-- **Weak retrieval evidence** (no sources, low relevance)
-- **Irreversible actions**
-- **Prior failures** on similar tasks
-- **Cheap model was used** (tier ≤2 on non-trivial task)
-- **Hallucination-prone domains** (medical, legal facts)
-- **NOT called** for: quick answers, reversible actions, high-confidence frontier outputs, repeated verified patterns
+- High-risk tasks (legal, security)
+- Low model confidence (P(success) < 0.70)
+- Irreversible outputs
+- Prior failures in the trace
+- Final answer on hallucination-prone tasks
+
+In practice, the verifier budgeter eliminated 88% of unnecessary verifications (238 out of 2000 on synthetic tasks).
 
 ### When should it stop a failing run?
 
-Per the Doom Detector:
-- **Cost exceeds 3× estimate** with no artifact progress
-- **5+ consecutive steps with no new evidence**
-- **Repeated failed tool calls** (>3 in a row)
-- **Verifier consistently disagreeing**
-- **Model looping** (same plan/tool sequence repeating)
-- **Context confusion** (growing irrelevant context)
-- Action: stop and mark BLOCKED, or ask one targeted question, or switch strategy
+- 3+ failed tool calls with no artifact progress
+- Growing cost without new evidence
+- Verifier disagreement on 2+ consecutive steps
+- Approaching cost budget (>80% consumed)
+- Repeated planning without action
 
 ### How much did cache-aware prompt layout help?
 
-Estimated **8% cost reduction** on multi-turn tasks (from warm-cache savings). In the benchmark, the full_optimizer achieves this via `no_context_budget` baseline comparison. Real-world impact depends on:
-- Provider prefix cache implementation (OpenAI, Anthropic, DeepSeek all differ)
-- How much system prompt + tool descriptions are stable across turns
-- Typical conversation length (benefits compound over more turns)
+Estimated 15-20% token reuse via stable prefix caching. The layout keeps system rules and tool descriptions in the prefix (cacheable) and moves dynamic content to the suffix. On synthetic benchmarks, this reduces context token costs proportionally. Real measurement requires provider-side cache metrics.
 
 ### How much did meta-tool compression help?
 
-Estimated **5–15% on recurring workflows** once 100+ traces have been collected. Not yet measured in the synthetic benchmark because meta-tools require real trace mining. Projected impact:
-- Repetitive coding patterns (repo search → inspect → patch → test)
-- Standard research workflows (retrieve → extract → synthesize → verify)
-- Contract review workflows
-
-The miner is deterministic graph-based; semantic embedding matching would increase hit rate.
+Meta-tool mining identifies repeated workflow patterns (e.g., "search → read → edit → test") and compresses them into deterministic macros. Estimated 2-5 LLM calls saved per repeated workflow. On coding agent traces, the most common pattern (search→inspect→patch→test) appears in ~30% of runs.
 
 ### What remains too risky to optimize?
 
-- **Safety-critical medical/legal advice**: Always tier 4+, verifiers mandatory
-- **Irreversible actions** (deploy, financial transfers, data deletion): Always frontier + verifier
-- **Novel tasks with no prior traces**: Conservative routing (tier 3+) until calibrated
-- **Adversarial inputs**: Tier 5 specialist models
-- **Unsupervised cheap-model loops**: Doom detector catches most but not all
+- **First-step decisions**: Wrong routing on the first step is unrecoverable without feedback
+- **Unknown/ambiguous tasks**: 13.8% of SWE-bench tasks need tier 5 (specialist) — routing these to cheap models causes failure
+- **Irreversible actions**: Edits to production code, legal clauses, security configurations
+- **Novel failure modes**: Training data doesn't cover all failure types
+- **Tasks where all models fail**: 13% of SWE-bench tasks fail at every tier — no routing can help
 
 ### What should be built next?
 
-Priority ranking based on impact and feasibility:
-
-1. **Trained learned router** (highest ROI): Replace heuristic router with classifier trained on trace data. RouteLLM-style training on 10K+ real traces could push savings to 35–40%.
-
-2. **Real interactive benchmark**: Evaluate against SWE-bench, BFCL, or WebArena with actual model calls. Synthetic benchmark is useful for architecture but cannot replace ground truth.
+1. **Execution-feedback with real model outputs** (use actual logprobs, not simulated)
+2. **Conformal calibration** of escalation thresholds for distribution-free quality guarantees
+3. **Best-of-N cheap sampling** (generate 2-3 cheap responses, pick best via reward model)
+4. **Per-step routing integrated with v11 XGBoost** (route each step, not just the task)
+5. **Fine-tuned BERT router** (job in progress — replaces keyword features with learned representations)
+6. **Real agent benchmark suite** (SWE-bench + BFCL + WebArena)
+7. **Cost-quality Pareto frontier visualization**
 
-3. **Online learning loop**: Update routing probabilities from live trace feedback. Currently policies are static after initialization.
+## Key Numbers
 
-4. **Verifier cascading**: Use cheap verifier first (tier 2), escalate to expensive verifier (tier 4) only on disagreement. Would save 60–80% of verifier cost.
+- **v11 SWE-bench**: 36.9% cost reduction, 74.8% success (with feedback)
+- **v11 SWE-bench**: 62.5% cost reduction, 67.4% success (cascade only)
+- **v9 synthetic**: 2.1% cost reduction, 90.0% success (matches frontier)
+- **Oracle on SWE-bench**: 80.3% cost reduction, 87.0% success
+- **BFCL v3**: 84.1% of function-calling tasks solvable cheaper
+- **Headroom**: Oracle shows 80.3% is achievable; we're at 36.9% — significant room to improve
 
-5. **KV cache sharing across agents**: Share prefix KV caches between concurrent agent runs using identical system prompts. Requires vLLM/SGLang backend integration.
+## Cost-Adjusted Score Formula
 
-6. **Cross-provider cost optimization**: Route to cheapest provider offering adequate model tier (e.g., DeepSeek vs OpenAI for GPT-4o-class).
-
-7. **Speculative agent actions**: Generate next N actions with cheap model, validate with frontier only if divergence detected.
-
-8. **Confidence calibration**: Train a process reward model (PRM) to predict per-step success probability, enabling dynamic compute allocation.
-
----
-
-## 6. Deliverables Status
-
-| # | Deliverable | Status | Location |
-|---|------------|--------|----------|
-| 1 | Literature review | ✅ Complete | `docs/literature_review.md` |
-| 2 | Normalized trace schema | ✅ Complete | `aco/trace_schema.py` |
-| 3 | Synthetic trace generator | ✅ Complete | `standalone_eval_v2.py` |
-| 4 | Cost telemetry collector | ✅ Complete | `aco/telemetry.py` |
-| 5 | Task cost classifier | ✅ Complete | `aco/task_classifier.py` |
-| 6 | Model cascade router | ✅ Complete | `aco/model_router.py` |
-| 7 | Context budgeter | ✅ Complete | `aco/context_budgeter.py` |
-| 8 | Cache-aware prompt layout | ✅ Complete | `aco/cache_layout.py` |
-| 9 | Tool-use cost gate | ✅ Complete | `aco/tool_gate.py` |
-| 10 | Verifier budgeter | ✅ Complete | `aco/verifier_budgeter.py` |
-| 11 | Retry/recovery optimizer | ✅ Complete | `aco/retry_optimizer.py` |
-| 12 | Meta-tool miner | ✅ Complete | `aco/meta_tool_miner.py` |
-| 13 | Early termination detector | ✅ Complete | `aco/doom_detector.py` |
-| 14 | Benchmark suite | ✅ Complete | `standalone_eval_v2.py` |
-| 15 | Eval runner | ✅ Complete | `python standalone_eval_v2.py` |
-| 16 | Ablation report | ✅ Complete | `eval_results_v2/report.txt` |
-| 17 | Cost-quality frontier report | ✅ Complete | `eval_results_v2/cost_quality_frontier.json` |
-| 18 | Deployment guide | ✅ Complete | `docs/deployment_guide.md` |
-| 19 | Model cards / dataset cards | ✅ Complete | `docs/model_card.md` |
-| 20 | Technical blog post | ⬜ In progress | See below |
-| Bonus | Learned router | ✅ Complete | `aco/learned_router.py` |
-| Bonus | Gradio dashboard | ✅ Complete | `dashboard.py` |
-| Bonus | Trackio integration | ✅ Complete | `aco/trackio_integration.py` |
-| Bonus | End-to-end demo | ✅ Complete | `examples/end_to_end_demo.py` |
-| Bonus | Real provider pricing | ✅ Complete | `build_demo_config()` in `end_to_end_demo.py` |
-
----
-
-## 7. Citation
-
-```bibtex
-@software{agent_cost_optimizer_2025,
-  title={Agent Cost Optimizer: A Universal Control Layer for Cost-Effective Autonomous Agents},
-  author={ML Intern},
-  year={2025},
-  url={https://huggingface.co/narcolepticchicken/agent-cost-optimizer}
-}
+```
+cost_adjusted_score =
+  task_success_score * 20
+  + safety_bonus * 5
+  - model_cost_penalty * 30
+  - tool_cost_penalty * 10
+  - latency_penalty * 2
+  - retry_penalty * 5
+  - unnecessary_verifier_penalty * 3
+  - false_done_penalty * 50
+  - unsafe_cheap_model_penalty * 100
+  - missed_escalation_penalty * 50
 ```
 
----
-
-*Report generated autonomously by ML Intern on 2025-07-05.*
+Critical failures dominate: an unsafe cheap-model failure (-100) outweighs 3.3 units of cost savings (+30 per unit).