docs/REWARD_HACKING.md

Reward Hacking Defense

Threat model

GRPO optimizes a policy directly against the scalar reward signal, so any exploitable gap between "what the reward measures" and "what the task actually requires" becomes a high-gradient attractor — the policy will collapse into the cheapest hack the verifier cannot see. Because our stack is verifier-style (Stim ground truth + PyMatching reference frame + a text parser), every reward must be a physical invariant or a cross-checkable auxiliary, not a regex of the model's own output.

The 5-reward composite

All five rewards are pure functions (parsed_action, sample, layout) -> float in [0, 1] evaluated independently and combined as a weighted sum clamped to [0, 1].

Name	Weight	What it rewards	What it cannot reward
logical_correction	0.40	Predicted Pauli frame, when applied at end-of-circuit, induces the same logical-Z observable flip Stim recorded as ground truth.	Anything not derivable from Stim's observable trace. Cannot be reverse-engineered from the prompt alone.
syndrome_consistency	0.20	Hamming similarity between predicted final-round detector parities (induced by the predicted X errors) and the observed final-round detector parities.	Earlier-round detectors are intentionally unscored; partial credit on bit-flipped hallucinations.
hamming_overlap	0.20	Mean of set-aware Jaccard(X_pred, X_ref) and Jaccard(Z_pred, Z_ref) against PyMatching's reference Pauli frame.	Symmetric "predict-empty-when-empty" hacks: the set-aware rule scores 0.0 for missed errors and 0.0 for false alarms.
format_compliance	0.10	1.0 only when the strict canonical X_ERRORS=[...] / Z_ERRORS=[...] form parses BOTH lists cleanly (lenient/partial parses score 0.5; nothing parseable scores 0.0).	Cannot be earned by whitespace tricks alone — the parser validates that every integer is in [0, num_data_qubits) and de-duplicates.
pymatching_beat	0.10	1.0 iff PyMatching got this syndrome wrong AND the model got it right.	Imitation of PyMatching: matching its output exactly forfeits the bonus on every syndrome PyMatching also gets right (most of them).

Weights sum to 1.00. Source of truth: openenv.yaml and qubit_medic.config.REWARD_WEIGHTS. Implementations live in qubit_medic/server/rewards.py.

Attack/defense matrix

Hack the model could attempt	Channel(s) that catch it
Output empty string	format_compliance = 0 (no strict pattern, no lists)
Memorize one canonical Pauli frame across all syndromes	hamming_overlap drops on novel syndromes (Jaccard against per-syndrome PyMatching reference); logical_correction drops to chance
Match PyMatching exactly on every shot	pymatching_beat = 0 whenever PyMatching is also correct (which is most syndromes), so the 0.10 channel never fires
Output a random valid format string	logical_correction collapses to ~chance; syndrome_consistency and hamming_overlap both drop
Skip syndrome reasoning, copy the in-prompt example block	The parser slices from the LAST X_ERRORS= key (so the prompt's example doesn't win); syndrome_consistency then penalises the stale answer
Game the format checker with whitespace / capitalisation tricks	format_compliance is parseability-based: _parse_int_list rejects out-of-range integers, drops dups, and strict_format requires the canonical =[...] form for the full 1.0
Inject extra correction operators ("over-correct")	hamming_overlap uses set-aware Jaccard whose union grows with false alarms (precision-aware), so over-correction strictly lowers the score
Predict an empty frame when the syndrome is non-empty (FIX 1, 2026-04)	syndrome_consistency is capped at 0.5 when prediction is empty AND any detector fired — the empty-everywhere collapse mode can never reach the full 1.0
Output a logical-flipped Pauli frame that coincidentally satisfies final-round parities	logical_correction = 0 because the implied observable flip differs from Stim ground truth; hamming_overlap also drops vs PyMatching's reference frame
Hallucinate qubit IDs outside [0, num_data_qubits) to spoof a long answer	_parse_int_list drops out-of-range tokens and flags parse_success=False, so format_compliance collapses to 0.0/0.5
Exploit per-axis Jaccard (predict X right, Z empty when Z is empty)	The set-aware rule (true_set empty AND pred_set empty -> 1.0; either non-empty asymmetric -> 0.0) plus the 0.5 mean across axes prevents winning by guessing one axis is empty
Time-stall (delay step beyond EPISODE_TIMEOUT_SECONDS) to evade scoring	The env builds a zero-reward RewardBreakdown and marks the episode truncated=True, so timeouts strictly hurt

Hard guarantees

These are physical invariants that hold by construction; no policy can satisfy them via parser games:

• Logical-Z preservation (Stim ground truth). predicted_observable_flip re-applies the predicted X errors as a Pauli frame at end-of-circuit and computes the implied flip on the logical Z observable. logical_correction is 1.0 iff the implied flip equals sample.actual_observable_flip recorded by Stim. There is no way to fake this without genuinely solving the decoder task on this syndrome.
• Final-round detector arithmetic. _syndrome_from_pauli_frame computes the implied final-round detector bits from the predicted X errors and the detector-to-data-qubit incidence map (final_detector_supports, derived from Euclidean adjacency in the rotated memory_z layout). These bits are compared against sample.syndrome_bits directly — the model never sees the comparison target.
• PyMatching reference frame. sample.pymatching_x_errors / pymatching_z_errors are computed by the Sparse Blossom matching decoder (PyMatching v2) for this exact syndrome; the model has no access to them at action time.
• Hidden ground truth. DecoderState carries true_x_errors, true_z_errors, actual_observable_flip, pymatching_observable_pred, circuit_text, and dem_text, but the externally-visible state() endpoint deliberately omits all of these (see qubit_medic/server/environment.py state() method). Only the reward functions see them.
• LLM-space → Stim-space conversion. Predicted qubit ids are mapped from LLM-space (0..N-1, the only IDs the prompt advertises) into Stim's internal coordinate system before scoring (layout.llm_to_stim). The model can't gain anything by guessing Stim's internal numbering.
• Episode pairing enforcement. step() raises a clean ValueError for unknown episode IDs (compliance audit 2026-04). A trainer cannot replay step() against a stale episode to harvest a stale reward.

Known weaknesses

Honest accounting of what this composite still does not catch:

• Hamming-similarity is not a strict equality on syndrome consistency. A predicted Pauli frame whose final-round implied bits happen to overlap the observed bits on most positions (without being correct) still scores partial credit on syndrome_consistency. The 0.5 cap on empty-prediction-vs-active-syndrome closes the worst case, but a near empty answer that flips one well-chosen qubit can still earn a high consistency score on prompts where most final-round detectors quiesced.
• hamming_overlap treats the PyMatching reference as ground truth. PyMatching is itself near-optimal but not optimal; on syndromes where the Stim-true correction differs from PyMatching's, a model that found the true correction is penalised on Reward 3 even though it's right on Reward 1. We accept this trade-off because Reward 5 (pymatching_beat) is the channel that explicitly rewards out-performing PyMatching, and it has its own 0.10 weight.
• No per-round detector scoring. Earlier-round detectors carry signal the LLM could exploit, but we score only the final round to keep the Pauli-frame action space tractable. A model could in principle "ignore intermediate rounds" without penalty as long as its terminal frame is correct — which is the same trade-off AlphaQubit made.
• Format compliance is binary-ish (0 / 0.5 / 1). The 2026-04 spec rewrite removed full credit for non-canonical resemblances; this is intentional, but it means a model that emits beautiful chain-of-thought reasoning and then forgets the final X_ERRORS=[...] line gets reduced credit identical to a near-miss. We trade interpretability for anti-gaming.
• pymatching_beat is sparse. Most syndromes are easy and PyMatching wins; the bonus only fires on the hard tail. This is by design (the trajectory of its mean is the proof of post-imitation behaviour) but it means GRPO sees this signal as ~zero-sum noise for most of training.
• No protection against constant-stream prompts. If a future trainer modification kept episodes alive across step() calls, the active-episode bookkeeping could in principle leak via observed reward statistics. The current single-step-per-episode design (done=True after every step) prevents this; do not relax it without a fresh hacking-surface review.