docs/20260511_reader.md

Abstract-CoT on Qwen3-4B — Reader Report

What we ran

Three policy-iteration (PI) rounds of the paper's Abstract-CoT warm-up on Qwen3-4B / MATH-500. The idea: replace the model's normal verbal chain-of-thought with a short sequence of tokens from a reserved 64-symbol "abstract" vocabulary V_abs = {<TOKEN_A>, ..., <TOKEN_BL>}. The model is taught to use this short discrete trace as its "reasoning scratchpad" before emitting the answer.

prompt  ─►  <beginabstract>  z_1 ... z_m  <endabstract>  answer
            └─────── z̃ ∈ V_abs^m, m ≤ 128 ───────┘

Each PI round has two phases, both trained as standard SFT against the Dolci-Think dataset:

• Phase A — Bottleneck SFT. Train on the packed sequence [prompt; verbal-CoT; z̃; answer] with a custom attention mask: the answer is forbidden from attending to the verbal CoT. So any information from the CoT that the answer needs must flow through z̃. This is what teaches the abstract trace to act as a compressed reasoning bridge.

  • Round 1: z̃ is just random V_abs tokens (no signal yet).
  • Rounds 2+: z̃ is sampled on-policy from the previous round's model, conditioned on (prompt, CoT). Now the bottleneck has a real teacher signal.

• Phase B — Self-distillation. Train on [prompt; z̃; answer] with normal causal attention, where z̃ is now generated from the prompt alone (no CoT in context). This teaches the model to produce the abstract trace directly from the question, not just consume one provided by a teacher.

After T=3 rounds of A→B→A→B→A→B, the model can in principle take a math problem, emit ~20 abstract tokens, then emit the answer — using the abstract trace as its only thinking.

Results

Run	MATH-500 acc	Mean tokens
Paper Baseline (verbal CoT)	83.2	1087
Our Baseline (Qwen3-4B, verbal CoT)	84.60	1045
Paper Abstract-CoT Warm-up	86.2	168
Prior smoke (T=1, 5k, 1 epoch)	73.20	433
This run (T=3, 5k, 1 epoch)	72.00	432

GPU resources — this run vs. prior smoke vs. paper

	This run	Prior smoke (paper repo)	Paper
GPUs for SFT	2× A100-SXM4-80GB	2× A100-SXM4-40GB	8× H100-80GB
Total VRAM	160 GB	80 GB	640 GB
Total GPU-hours used	~22 GPU-hr (11 hr × 2)	~2 GPU-hr (1 hr × 2)	not stated, RL adds 32× H100
Training memory headroom	comfortable at seq 8k, LoRA	smoke hit ZeRO-3 OOM at seq>1024	full FT @ seq ~16k probably

What having 80 GB cards unlocked vs. the smoke's 40 GB:

1. seq_len 8192 for Phase A (vs the smoke's 2048). At 2k, 98% of Dolci CoTs got truncated from the right, so the bottleneck mostly compressed a problem statement. At 8k, ~60% of CoTs fit fully and the actual reasoning makes it into training.
2. vLLM at TP=2 for inference without idling a card. The smoke ran on 3× 40GB; Qwen3-4B has 32 heads so TP must divide 32, forcing TP=2 with one GPU idle during eval. On our 2-GPU box, both cards work.
3. Full fine-tuning is now feasible (paper's path). Adam fp32 states for the full 4.86 B params (~40 GB) fit in ZeRO-3 across 2× 80GB without needing CPU offload. We didn't use this — kept LoRA per the smoke's recommendation for the 12 hr budget — but it's the obvious next experiment, and it would not have been possible on the smoke hardware.

What we didn't get from the bigger cards: per-step training speed. Both A100-40GB and A100-80GB are the same GA100 silicon — same SM count, same clock. The 80GB version has 25% more HBM bandwidth, which slightly helps inference (memory-bound) but not training (compute-bound on bf16 tensor cores). Measured per-step time at seq 2k matched the smoke (12 s/step); at seq 8k we measured 72 s/step (5.7× slower due to attention scaling quadratically).

What the numbers tell us

Baseline reproduces the paper. Our 84.6 vs paper 83.2 is within evaluation noise — the verbal-CoT pipeline is calibrated correctly.

Three rounds didn't beat one. This is the most interesting finding. The paper's signature result is that on-policy iteration (T=3) lifts accuracy substantially over T=1. We saw no lift: 72.0 vs 73.2 is within the temp=0.7 sampling noise of the abstract trace stage.

But the optimizer was clearly working. Each round started from a lower loss than the previous one:

Round	Phase A start loss	Phase B start loss
1 (random abstract traces)	3.49	0.49
2 (on-policy traces from round 1)	0.35	0.29
3 (on-policy traces from round 2)	0.27	0.21

The model is absorbing the on-policy abstract traces — they're carrying real signal by round 3. That signal just isn't translating to MATH-500 accuracy.

Token count converged. Mean total tokens stayed flat at ~432 (vs paper's 168). The model isn't learning to be as terse as the paper's, but it's also not bloating responses round-over-round.

Why T=3 didn't lift accuracy — most-likely explanations

1. LoRA is the binding constraint. With ~17% of parameters trainable (mostly the new abstract-token embeddings, plus rank-32 adapters), the model's existing "answer-from-prompt" reflex is too strong for a short discrete bottleneck to redirect. The paper does full fine-tuning, where the existing math knowledge can be reshaped to actually route through the abstract trace. We can't reshape with LoRA — we can only nudge.

2. 5k samples is far below the paper's 600k. On-policy iteration needs enough novel (problem, CoT) pairs that each round produces meaningfully different abstract traces. At 5k, rounds 2 and 3 mostly revisit the same examples with mild trace variations — not much new pressure on the bottleneck.

3. Eval stochasticity. The abstract trace is sampled at temperature 0.7 (the m_min=16 length floor requires some sampling). Run-to-run variance on N=500 examples is 1–2 points. Both 73.2 (smoke) and 72.0 (this run) are within that band.

4. Possible counterintuitive seq_len effect. We bumped max sequence length from 2048 to 8192 so the model could see full CoTs during Phase A bottleneck training. But the bottleneck's "quality" depends on the gap between what reaches the abstract trace and what the answer can reconstruct from prompt alone. Letting more CoT into the training context may have made it easier for the model to learn shortcuts that don't depend on Z̃. Worth ablating.

What would move the number

In rough order of expected lift:

1. Full fine-tuning instead of LoRA. This is the smoke report's #1 recommendation and our experience confirms it — the paper's gap is most plausibly explained by representation capacity, not data scale.
2. More data (30k–100k). Probably needed to actually exploit T=3.
3. More epochs per phase (paper uses 3, we used 1).

Of these, full-FT at 5k is the cheapest diagnostic — if it lifts accuracy substantially, data scaling is the next lever; if not, we know the issue is elsewhere (e.g., reward shaping, which the paper's RL stage handles but we skipped).

Bottom line

The pipeline works end-to-end and reproduces the smoke result. The on-policy iteration is doing what it's supposed to mechanically (per the per-round loss curves), but the LoRA + 5k-data regime caps the accuracy delta at noise level. The gap to the paper's 86.2 is real and consistent with the known scope reductions (LoRA, 1/120 the data, 1/3 the epochs, no RL).