Upload ABLATION_REPORT_v2.md

ABLATION_REPORT_v2.md

@@ -0,0 +1,79 @@
+# Speculative Tool Actions: Ablation Report (v2)
+
+## Experiment Design
+
+We test whether speculative decoding can be adapted from token prediction to agent action prediction.
+
+**Task:** Given a conversation context (multi-turn user/assistant/tool), predict the next agent action from 9 classes: `tool_call`, `retrieval`, `file_read`, `file_write`, `repair`, `verifier`, `ask_clarification`, `final_answer`, `BLOCKED`.
+
+**Models:**
+- **Cheap proposer:** Qwen3-1.7B + LoRA fine-tuned for 2 epochs on synthetic action-prediction data (96.7% train accuracy)
+- **Strong model:** Qwen3-8B frozen (zero-shot)
+- **Reward model:** Qwen3-4B + LoRA trained on preference pairs from which-actions-are-correct (70% train accuracy)
+
+**Configurations:**
+
+| Config | Description | Expected Cost (normalized) |
+|--------|-------------|---------------------------|
+| A | Strong 8B model only | 1.00 |
+| B | Cheap 1.7B proposer only | 0.15 |
+| C | Cheap proposes → 8B verifies (ACCEPT/REJECT) | 0.25–1.25 |
+| D | Cheap proposes → 4B reward model scores | 0.25 |
+| E | Cheap generates 3 proposals → reward model picks best | 0.75 |
+
+## Results (200 eval examples, dual A10G)
+
+| Config | Accuracy | Cost | xRandom (0.111) | xMajority (0.240) | Accept Rate |
+|--------|----------|------|-----------------|-------------------|-------------|
+| A (8B) | **0.400** | 1.000 | 3.6× | 1.7× | — |
+| B (1.7B) | **0.510** | 0.150 | 4.6× | 2.1× | — |
+| C (C+S) | **0.400** | 1.250 | 3.6× | 1.7× | 0.000 |
+| D (C+R) | **0.510** | 0.250 | 4.6× | 2.1× | 0.215 |
+| E (multi) | pending | — | — | — | — |
+
+### Key Findings
+
+**1. Cheap model outperforms strong model (B > A: 51% vs 40%)**
+
+The fine-tuned 1.7B proposer learned the task from synthetic data. The frozen 8B model, despite 4.7× more parameters, had zero training on the task and defaults to generic completions. **Fine-tuning matters more than scale for this task.**
+
+**2. Verifier (C): 0% accept rate — completely useless**
+
+The 8B ACCEPT/REJECT verifier rejected EVERY proposal from the cheap model, then fell back to its own (worse) predictions. Result: same accuracy as A (40%) but higher cost (1.25 vs 1.00). The verifier is either:
+- Telling the truth (it genuinely believes all proposals are wrong)
+- Collapsed to always outputting "REJECT"
+
+Either way, it provides zero benefit.
+
+**3. Reward model (D): 21.5% accept rate, all scores negative (mean -1.52)**
+
+The 4B reward model scores all proposals negatively, with only ~1/5 above the acceptance threshold of -1.0. But since Config B and D have identical accuracy (51%), the reward model is not effectively filtering — it's just adding cost without changing outcomes. The accepted proposals are no more accurate than the rejected ones.
+
+**4. Cost-quality frontier is degenerate**
+
+The only sensible point on the frontier is **Config B**: 51% accuracy at 0.15× cost. Configs A, C, and D are all dominated — they cost more and deliver equal or worse accuracy.
+
+## Why This Task is Challenging
+
+The eval dataset uses **real conversational messages** (from synthetic templates but in natural chat format), while the proposer was fine-tuned on **"Action: <type>\n<reason>" format**. The training/eval distribution mismatch means:
+
+- The proposer learned to generate `Action: tool_call\nReason: ...` when given `"Predict the next action for:\n\nuser: ... assistant: ..."`
+- The 8B model had no such training and produces natural language continuations instead
+- The verifier was trained on preference pairs, not on ACCEPT/REJECT decisions
+
+**The speculative decoding architecture cannot fix a weak signal problem.** When the task itself is poorly learned, adding complex routing (verification, reranking) only adds cost without improving accuracy.
+
+## Recommendations
+
+1. **Fix the training data:** Generate SFT data using the same chat-template format as the eval dataset. Use `tokenizer.apply_chat_template()` consistently.
+2. **Train the strong model:** Fine-tune Qwen3-8B on the same SFT data for a fair comparison of proposer size.
+3. **Train the verifier properly:** The ACCEPT/REJECT verifier needs training examples showing correct proposals paired with ACCEPT decisions.
+4. **Use a real agent benchmark:** The synthetic action prediction task has limited ecological validity. A task like BFCL (Berkeley Function Calling Leaderboard) would provide realistic tool-calling traces.
+
+## Deliverables
+
+- **Dataset:** `narcolepticchicken/speculative-actions-eval` (500 conversational examples with action labels)
+- **Proposer:** `narcolepticchicken/speculative-proposer-qwen3-1.7b` (LoRA adapter)
+- **Verifier:** `narcolepticchicken/speculative-verifier-qwen3-4b` (LoRA adapter for reward model)
+- **Eval runner:** `narcolepticchicken/speculative-tool-actions` (eval scripts + results)
+- **Results:** `narcolepticchicken/speculative-tool-actions/eval_results_v2.json`