Upload docs/REWARD_HACKING.md with huggingface_hub

docs/REWARD_HACKING.md

@@ -0,0 +1,120 @@
+# Reward Hacking Defense
+
+## Threat model
+
+GRPO optimizes a policy directly against the scalar reward signal, so any
+exploitable gap between "what the reward measures" and "what the task
+actually requires" becomes a high-gradient attractor — the policy will
+collapse into the cheapest hack the verifier cannot see. Because our stack
+is verifier-style (Stim ground truth + PyMatching reference frame + a text
+parser), every reward must be a *physical invariant* or a *cross-checkable
+auxiliary*, not a regex of the model's own output.
+
+## The 5-reward composite
+
+All five rewards are pure functions
+`(parsed_action, sample, layout) -> float in [0, 1]` evaluated independently
+and combined as a weighted sum clamped to `[0, 1]`.
+
+| Name | Weight | What it rewards | What it cannot reward |
+|------|--------|-----------------|-----------------------|
+| `logical_correction` | 0.40 | Predicted Pauli frame, when applied at end-of-circuit, induces the same logical-Z observable flip Stim recorded as ground truth. | Anything not derivable from Stim's observable trace. Cannot be reverse-engineered from the prompt alone. |
+| `syndrome_consistency` | 0.20 | Hamming similarity between *predicted* final-round detector parities (induced by the predicted X errors) and the *observed* final-round detector parities. | Earlier-round detectors are intentionally unscored; partial credit on bit-flipped hallucinations. |
+| `hamming_overlap` | 0.20 | Mean of set-aware Jaccard(X_pred, X_ref) and Jaccard(Z_pred, Z_ref) against PyMatching's reference Pauli frame. | Symmetric "predict-empty-when-empty" hacks: the set-aware rule scores 0.0 for missed errors and 0.0 for false alarms. |
+| `format_compliance` | 0.10 | 1.0 only when the strict canonical `X_ERRORS=[...] / Z_ERRORS=[...]` form parses BOTH lists cleanly (lenient/partial parses score 0.5; nothing parseable scores 0.0). | Cannot be earned by whitespace tricks alone — the parser validates that every integer is in `[0, num_data_qubits)` and de-duplicates. |
+| `pymatching_beat` | 0.10 | 1.0 iff PyMatching got this syndrome wrong AND the model got it right. | Imitation of PyMatching: matching its output exactly forfeits the bonus on every syndrome PyMatching also gets right (most of them). |
+
+Weights sum to 1.00. Source of truth: `openenv.yaml` and
+`qubit_medic.config.REWARD_WEIGHTS`. Implementations live in
+`qubit_medic/server/rewards.py`.
+
+## Attack/defense matrix
+
+| Hack the model could attempt | Channel(s) that catch it |
+|---|---|
+| Output empty string | `format_compliance = 0` (no strict pattern, no lists) |
+| Memorize one canonical Pauli frame across all syndromes | `hamming_overlap` drops on novel syndromes (Jaccard against per-syndrome PyMatching reference); `logical_correction` drops to chance |
+| Match PyMatching exactly on every shot | `pymatching_beat = 0` whenever PyMatching is also correct (which is most syndromes), so the 0.10 channel never fires |
+| Output a random valid format string | `logical_correction` collapses to ~chance; `syndrome_consistency` and `hamming_overlap` both drop |
+| Skip syndrome reasoning, copy the in-prompt example block | The parser slices from the LAST `X_ERRORS=` key (so the prompt's example doesn't win); `syndrome_consistency` then penalises the stale answer |
+| Game the format checker with whitespace / capitalisation tricks | `format_compliance` is parseability-based: `_parse_int_list` rejects out-of-range integers, drops dups, and `strict_format` requires the canonical `=[...]` form for the full 1.0 |
+| Inject extra correction operators ("over-correct") | `hamming_overlap` uses set-aware Jaccard whose union grows with false alarms (precision-aware), so over-correction strictly lowers the score |
+| Predict an empty frame when the syndrome is non-empty (FIX 1, 2026-04) | `syndrome_consistency` is **capped at 0.5** when prediction is empty AND any detector fired — the empty-everywhere collapse mode can never reach the full 1.0 |
+| Output a logical-flipped Pauli frame that *coincidentally* satisfies final-round parities | `logical_correction = 0` because the implied observable flip differs from Stim ground truth; `hamming_overlap` also drops vs PyMatching's reference frame |
+| Hallucinate qubit IDs outside `[0, num_data_qubits)` to spoof a long answer | `_parse_int_list` drops out-of-range tokens and flags `parse_success=False`, so `format_compliance` collapses to 0.0/0.5 |
+| Exploit per-axis Jaccard (predict X right, Z empty when Z is empty) | The set-aware rule (`true_set` empty AND `pred_set` empty -> 1.0; either non-empty asymmetric -> 0.0) plus the 0.5 mean across axes prevents winning by guessing one axis is empty |
+| Time-stall (delay step beyond `EPISODE_TIMEOUT_SECONDS`) to evade scoring | The env builds a zero-reward `RewardBreakdown` and marks the episode `truncated=True`, so timeouts strictly hurt |
+
+## Hard guarantees
+
+These are physical invariants that hold by construction; no policy can
+satisfy them via parser games:
+
+- **Logical-Z preservation (Stim ground truth).** `predicted_observable_flip`
+  re-applies the predicted X errors as a Pauli frame at end-of-circuit and
+  computes the implied flip on the logical Z observable. `logical_correction`
+  is 1.0 iff the implied flip equals `sample.actual_observable_flip` recorded
+  by Stim. There is no way to fake this without genuinely solving the decoder
+  task on this syndrome.
+- **Final-round detector arithmetic.** `_syndrome_from_pauli_frame` computes
+  the implied final-round detector bits from the predicted X errors and the
+  detector-to-data-qubit incidence map (`final_detector_supports`, derived
+  from Euclidean adjacency in the rotated memory_z layout). These bits are
+  compared against `sample.syndrome_bits` directly — the model never sees
+  the comparison target.
+- **PyMatching reference frame.** `sample.pymatching_x_errors` /
+  `pymatching_z_errors` are computed by the Sparse Blossom matching decoder
+  (PyMatching v2) for this exact syndrome; the model has no access to them
+  at action time.
+- **Hidden ground truth.** `DecoderState` carries `true_x_errors`,
+  `true_z_errors`, `actual_observable_flip`, `pymatching_observable_pred`,
+  `circuit_text`, and `dem_text`, but the externally-visible `state()`
+  endpoint *deliberately omits all of these* (see
+  `qubit_medic/server/environment.py` `state()` method). Only the reward
+  functions see them.
+- **LLM-space → Stim-space conversion.** Predicted qubit ids are mapped from
+  LLM-space (0..N-1, the only IDs the prompt advertises) into Stim's
+  internal coordinate system before scoring (`layout.llm_to_stim`). The
+  model can't gain anything by guessing Stim's internal numbering.
+- **Episode pairing enforcement.** `step()` raises a clean `ValueError` for
+  unknown episode IDs (compliance audit 2026-04). A trainer cannot replay
+  step() against a stale episode to harvest a stale reward.
+
+## Known weaknesses
+
+Honest accounting of what this composite still does **not** catch:
+
+- **Hamming-similarity is not a strict equality on syndrome consistency.**
+  A predicted Pauli frame whose final-round implied bits happen to overlap
+  the observed bits on most positions (without being correct) still scores
+  partial credit on `syndrome_consistency`. The 0.5 cap on
+  empty-prediction-vs-active-syndrome closes the worst case, but a *near*
+  empty answer that flips one well-chosen qubit can still earn a high
+  consistency score on prompts where most final-round detectors quiesced.
+- **`hamming_overlap` treats the PyMatching reference as ground truth.**
+  PyMatching is itself near-optimal but not optimal; on syndromes where the
+  Stim-true correction differs from PyMatching's, a model that found the
+  *true* correction is penalised on Reward 3 even though it's right on
+  Reward 1. We accept this trade-off because Reward 5 (`pymatching_beat`)
+  is the channel that explicitly rewards out-performing PyMatching, and it
+  has its own 0.10 weight.
+- **No per-round detector scoring.** Earlier-round detectors carry signal
+  the LLM could exploit, but we score only the final round to keep the
+  Pauli-frame action space tractable. A model could in principle "ignore
+  intermediate rounds" without penalty as long as its terminal frame is
+  correct — which is the same trade-off AlphaQubit made.
+- **Format compliance is binary-ish (0 / 0.5 / 1).** The 2026-04 spec
+  rewrite removed full credit for non-canonical resemblances; this is
+  intentional, but it means a model that emits beautiful chain-of-thought
+  reasoning *and then forgets the final `X_ERRORS=[...]` line* gets
+  reduced credit identical to a near-miss. We trade interpretability for
+  anti-gaming.
+- **`pymatching_beat` is sparse.** Most syndromes are easy and PyMatching
+  wins; the bonus only fires on the hard tail. This is by design (the
+  trajectory of its mean is the proof of post-imitation behaviour) but it
+  means GRPO sees this signal as ~zero-sum noise for most of training.
+- **No protection against constant-stream prompts.** If a future trainer
+  modification kept episodes alive across `step()` calls, the active-episode
+  bookkeeping could in principle leak via observed reward statistics. The
+  current single-step-per-episode design (`done=True` after every `step`)
+  prevents this; do not relax it without a fresh hacking-surface review.