# v0.5 Chat-Tune Postmortem (2026-05-03)

The canonical v0.5 chat is `chat-v3 (MCQ-tuned)` at **36.9%** on CTIBench MCQ.
This document records the recovery attempts that followed and what they
actually changed about our understanding.

## Result table

| Run | Recipe | Steps | LR | Val | CTIBench MCQ |
|---|---|---|---|---|---|
| chat-v2 | Cybersec Q&A only, no MCQ | 1500 | 5e-5 | — | 19.0% |
| **chat-v3 (canonical)** | **Raw letter-only MCQ × 5** | 1500 | 5e-5 | — | **36.9%** |
| chat-v4 (RAFT) | RAG-augmented chat-v3 mix | 1500 | 5e-5 | — | 25.0% |
| chat (v0.5 base re-tune) | chat-v3 recipe on v0.5 base | 1500 | 5e-5 | — | 32.5% |
| chat-long | chat-v3 mix, 4000 steps | 4000 | 5e-5 | — | 17.1% |
| chat-recovered | CoT MCQ × 1 + small-talk × 30 | 1500 | 3e-5 | 2.808 | 30.8% |
| chat-v4 (failed) | Hybrid + lr 2e-4 | 300 | 2e-4 | diverged | killed |
| **chat-v5 (this run)** | **Hybrid raw × 5 + CoT × 2 + small-talk × 8** | 2000 | 5e-5 | 2.990 | **34.8%** |

## What we learned

### What chat-v3 actually does

The 36.9% canonical is a *pattern-match shortcut*, not reasoning. With raw
letter-only MCQ at × 5 multiplier, the model learns "after the prompt ends in
'Answer:', emit a single letter consistent with the surface features of the
options." This is a known class of MCQ artifact (Answer Matching > MCQ,
arXiv 2507.02856) — sub-100M models can hit reasonable MCQ scores by
exploiting the choice distribution without understanding the question.

### Why CoT-MCQ alone made it worse

`chat-recovered` (30.8%) replaced the letter-only MCQ × 5 with CoT MCQ × 1.
The CoT records have the format `"B. <1-2 sentence justification>"` — Qwen-14B
generated the reasoning. The hypothesis, from Phi-3.5-mini and OpenMath-Mini,
was that reasoning supervision should outperform pattern-match supervision
even at low multipliers.

It didn't — at 36M params, the model can't compress 1-2 sentences of cybersec
reasoning into useful weight updates, and it loses the letter-shortcut signal
in the process. Documented size effect: weaker students benefit from coarser
supervision; long rationales over-smooth gradients (Skip-Thinking, arXiv
2505.18642; Unveiling Key Factors for Distilling CoT, arXiv 2502.18001).

The 30 × small-talk multiplier compounded the damage by pushing task-data
share below 5% of the SFT mix — well outside the SmolLM2 reference of
≥ 20% task share.

### Why chat-v4 (lr 2e-4) diverged

Research said an undertrained backbone needs aggressive SFT lr to escape a
bad pretrain basin. SmolLM2 uses 3e-4 SFT lr at 135M params. Scaled down to
36M with mean-init new tokens, 2e-4 was still too hot — val climbed
monotonically across 3 evals (3.175 → 3.285 → 3.403) before we killed it at
step 300.

Lesson: the SmolLM2 lr reference doesn't transfer linearly to 36M with new
embedding rows. The safe range is closer to 5e-5.

### What chat-v5 got right (and didn't)

The hybrid recipe (raw × 5 + CoT × 2, small-talk × 8, lr 5e-5, mean-init
embeddings) lifted the score from 30.8% → 34.8% — a real **+4.0 point** gain
over the prior recovery attempt. But it still trails canonical by 2.1 points.

The hybrid was directionally right — keeping the letter-shortcut anchor
(raw × 5) preserved the discriminative signal, while CoT × 2 added some
reasoning supervision without over-rotating. Mean-init for new tokens kept
the residual stream stable.

What it didn't fix: the letter-shortcut at × 5 is still doing most of the
work, and there's no mechanism in this recipe that actually transfers
*knowledge* into the model — only better calibration on top of the shortcut.
To beat 36.9% durably, the lever isn't another SFT recipe — it's either:

1. **Bigger model** (ghost-base ~350M) so reasoning supervision actually fits.
2. **Better pretrain coverage** of the CTIBench knowledge domain (more
   cyber threat intel, MITRE corpus depth) so the shortcut isn't the only
   path to a correct answer.
3. **Proper retrieval at inference** (RAG done right, not the chat-v4 RAFT
   attempt that conflated training-time and inference-time augmentation).

## Decision

- **Canonical stays:** `v0.5 chat-v3 (MCQ-tuned)` at 36.9% on the main HF repo.
- **Ship chat-v5 separately:** push to `Ghostgim/GhostLM-v0.5-experimental`
  with this postmortem in the model card. Honest framing: "improved CoT
  hybrid recipe, still 2.1pt below canonical, primarily of research interest."
- **No more chat-tune iterations on v0.5.** The 36.9% ceiling is a pretrain
  + capacity ceiling, not a recipe ceiling. Next swing should be ghost-base
  or a corpus-side fix, not another SFT permutation.

## Sources

- [Answer Matching Outperforms MCQ, arXiv 2507.02856](https://arxiv.org/abs/2507.02856)
- [Skip-Thinking, arXiv 2505.18642](https://arxiv.org/html/2505.18642v1)
- [Unveiling Key Factors for Distilling CoT, arXiv 2502.18001](https://arxiv.org/html/2502.18001v1)
- [SmolLM2, arXiv 2502.02737](https://arxiv.org/html/2502.02737v1)
- [How Abilities in LLMs are Affected by SFT Data Composition](https://openreview.net/forum?id=6M5G5hNiAU)