| # Quick Reasoning Rebuttal Check |
|
|
| This is a quick-turn held-out-task check for reasoning-heavy tasks. |
|
|
| - Model: `meta-llama/Llama-2-7b-chat-hf` dtype=fp16 device=cuda |
| - Tasks used for basis/eval: `gsm8k,commonsenseqa,strategyqa,arc_challenge,openbookqa,qasc,logiqa` |
| - Held-out tasks run: `gsm8k,logiqa` |
| - Per-task eval size: n_eval=32, n_subspace=64, layer=10 |
| - Protocol: LOTO heldout, forced_choice=True, do_sample=False |
|
|
| ## Per-task results |
|
|
| | Held-out | Type | n | Baseline | Decode-shared | Prefill-shared | Random | D-P delta | p | |
| |----------|-----------------------------------|----|--------------------|---------------|----------------|--------|---------------------|-------| |
| | gsm8k | Open-ended numeric reasoning | 32 | 0.0 | 0.0 | 0.0 | 0.0 | +0.0 [+0.0, +0.0] | 1 | |
| | logiqa | Logical reasoning multiple choice | 32 | 31.2 (chance 25.0) | 15.6 | 34.4 | 34.4 | -18.8 [-31.2, -6.2] | 0.036 | |
|
|
| ## Aggregate |
|
|
| - Mean accuracy: baseline=15.6, decode_shared=7.8, prefill_shared=17.2, random=17.2 |
| - Mean deltas vs baseline: decode=-7.8, prefill=+1.6, random=+1.6 |
| - Mean decode-minus-prefill delta: -9.4 |
| - Informative held-out tasks: `logiqa` |
| - Inconclusive due to baseline floor/chance: `gsm8k` |
|
|
| ## Interpretation |
|
|
| - `gsm8k` is currently inconclusive: baseline is at or near floor/chance, so this fold does not say much about decode-vs-prefill selectivity. |
| - `logiqa` is informative: decode-shared changes accuracy by -15.6 vs baseline and -18.8 vs prefill-shared. |
| - Use informative folds as rebuttal evidence that the decode-shared phenomenon is not confined to short classification tasks. |
|
|
