Write public PolyGuard project blog

blog.md

@@ -0,0 +1,1144 @@
+# PolyGuard OpenEnv: Training Medication-Safety Agents Inside a Verifier-Backed World
+
+Someone does not experience an unsafe medication regimen as "polypharmacy."
+They experience it as dizziness after a new sleep medication, bleeding after a
+painkiller is added to a blood thinner, confusion from a sedative-opioid
+combination, or a preventable emergency visit because five prescribers each saw
+one slice of the medication list.
+
+The dangerous part is often not a single drug. It is the combination: the wrong
+pair, the wrong dose in the wrong organ-function context, the missing lab, the
+duplicated class, the abrupt stop that should have been a taper, or the model
+that confidently says "looks fine" because it was never forced to act inside a
+safety-checked environment.
+
+That is the problem PolyGuard was built for.
+
+The [CDC medication-safety data page](https://www.cdc.gov/medication-safety/data-research/facts-stats/index.html)
+reports that adverse drug events send more than 1.5 million people to emergency
+departments in the United States every year, with almost 500,000
+hospitalizations. Adults 65 and older account for more than 600,000 of those
+visits. A CDC-authored [JAMA surveillance study](https://jamanetwork.com/journals/jama/fullarticle/2585977)
+found that older adults made up 34.5 percent of outpatient adverse-drug-event
+ED visits and had the highest hospitalization rate, 43.6 percent. Globally, the
+[WHO Medication Without Harm challenge](https://www.who.int/initiatives/medication-without-harm)
+estimates the cost associated with medication errors at USD 42 billion
+annually. AHRQ's deprescribing safety review summarizes estimates that
+[45 percent of older adults are exposed to polypharmacy and 58 percent to
+potentially inappropriate medications](https://www.ncbi.nlm.nih.gov/books/NBK600387/).
+
+Not every adverse drug event is caused by an incorrect drug combination. But
+these numbers describe the harm surface PolyGuard targets: medication decisions
+where combination risk, monitoring gaps, frailty, organ function, uncertainty,
+and action sequencing all matter at once.
+
+PolyGuard turns that problem into an OpenEnv-compatible reinforcement-learning
+environment for polypharmacy safety, medication optimization, deprescribing,
+safe substitution, missing-evidence recovery, and precision dosing. A language
+model policy observes a constrained patient/regimen state, chooses one legal
+candidate action, receives verifier-backed reward, and improves through SFT
+plus GRPO-style post-training.
+
+It is not medical software and it is not clinical advice. It is a controlled
+research environment for studying how language-model policies can be trained,
+audited, and stress-tested on safety-critical medication action selection.
+
+## What To Open First
+
+- GitHub repository:
+  [Vishwa-docs/Meta_Pytorch_OpenEnv_Scaler_VK](https://github.com/Vishwa-docs/Meta_Pytorch_OpenEnv_Scaler_VK)
+- Live product Space:
+  [TheJackBright/polyguard-openenv-workbench](https://huggingface.co/spaces/TheJackBright/polyguard-openenv-workbench)
+- One-run Colab/HF notebook:
+  [PolyGuard_SFT_GRPO_One_Run_Runner.ipynb](https://colab.research.google.com/github/Vishwa-docs/Meta_Pytorch_OpenEnv_Scaler_VK/blob/master/polyguard-rl/PolyGuard_SFT_GRPO_One_Run_Runner.ipynb)
+- Final evidence index:
+  [polyguard-rl/docs/results/final_submission_evidence/README.md](polyguard-rl/docs/results/final_submission_evidence/README.md)
+- Artifact and traceability guide:
+  [polyguard-rl/docs/submission_artifacts.md](polyguard-rl/docs/submission_artifacts.md)
+- Final artifact/evidence Space:
+  [adithya9903/polyguard-openenv-final-artifacts](https://huggingface.co/spaces/adithya9903/polyguard-openenv-final-artifacts)
+
+The final artifact/evidence Space hosts the Qwen 3B artifact bundle. The Qwen
+0.5B and 1.5B runs were trained using a second Hugging Face account, so their
+model artifacts could not be hosted in the same final Space. Their report
+mirrors are checked into this repo:
+[0.5B reports](polyguard-rl/docs/results/submission_evidence_qwen_0_5b_1_5b_3b/reports/runs/qwen-qwen2-5-0-5b-instruct)
+and
+[1.5B reports](polyguard-rl/docs/results/submission_evidence_qwen_0_5b_1_5b_3b/reports/runs/qwen-qwen2-5-1-5b-instruct).
+
+## The Research Bet
+
+Medication safety is combinatorial, partially observable, and high stakes. A
+useful policy has to do more than generate a plausible answer. It has to notice
+drug-drug interaction risk, reason about comorbidities and organ function,
+respect taper and monitoring requirements, choose safe substitutions, abstain
+or ask for review when uncertainty is high, and expose why it acted.
+
+The machine-learning pressure is just as real. If a medication vocabulary has
+500 drugs, the number of possible five-drug combinations is:
+
+```text
+C(500, 5) = 255,244,687,600
+```
+
+Exhaustive search is not a serious option. The paper that inspired this
+project, [Neural Bandits for Data Mining: Searching for Dangerous Polypharmacy](https://arxiv.org/abs/2212.05190),
+frames dangerous polypharmacy discovery as a bandit search problem over a huge
+combination space. It benchmarks neural bandit search over simulated
+polypharmacy datasets with 500 drugs and 100,000 distinct combinations, and
+reports detection of up to 72 percent of potentially inappropriate
+polypharmacies with 99 percent average precision after 30,000 time steps.
+
+PolyGuard borrows that search instinct, then moves the problem from offline
+combination mining into an agentic environment. The policy sees a patient
+state, chooses among legal clinical action candidates, and is judged by a
+deterministic verifier and reward router rather than by free-form text
+preference alone.
+
+The research question is narrow and concrete:
+
+Can environment-backed feedback make a small open model better at safe
+medication action selection than prompt-only, first-legal, rule-only, or
+single-agent baselines?
+
+The answer in this repository is an inspectable system:
+
+1. A finite-horizon OpenEnv simulation for medication decisions.
+2. A constrained action space, so the model chooses candidate actions instead
+   of inventing arbitrary clinical instructions.
+3. A legality verifier that prevents unsafe state mutation.
+4. Thirteen reward components rolled into four primary reward channels.
+5. A multi-agent policy stack with supervisor routing, contextual bandit
+   reranking, planner selection, critic veto, and explanation logging.
+6. SFT for format and clinical-prior warm start.
+7. GRPO with environment-backed reward, not an opaque LLM judge.
+8. Agentic evaluation with baseline comparison, policy ablations, post-save
+   inference, robustness checks, action traces, and failure mining.
+
+![PolyGuard system architecture](polyguard-rl/docs/assets/diagrams/system_architecture.png)
+
+## A Failure Trace That Motivated the Design
+
+In the final matched-seed traces, the failure mode is not abstract. On seeds
+`8000` and `8004`, the basic prompt-style proxy repeatedly chose `cand_01`,
+the first legal candidate. In those cases, `cand_01` meant `KEEP_REGIMEN` while
+a hidden `warfarin_like` + `nsaid_like` interaction remained unresolved. The
+verifier recorded `holdout_ddi_not_addressed`.
+
+The full PolyGuard pipeline selected `cand_03`, a safer intervention candidate,
+and avoided those failure reasons.
+
+That is the core argument of the project: medication AI should be judged inside
+a stateful safety environment, not only by whether its answer sounds clinically
+plausible.
+
+Internal evidence:
+[basic_llm_vs_polyguard_report.json](polyguard-rl/docs/results/final_submission_evidence/reports/basic_llm_vs_polyguard_report.json)
+and
+[action_traces.jsonl](polyguard-rl/docs/results/final_submission_evidence/reports/action_traces.jsonl).
+
+## Safety Contract
+
+PolyGuard does not let a model directly mutate a medication list from free
+text. Every decision is candidate-based, verifier-checked, reward-decomposed,
+and traced. Illegal actions can be scored, penalized, and logged, but they do
+not change patient state.
+
+The repo evidence for this contract is spread across the environment, rules,
+and final reports:
+
+| Claim | Repo evidence |
+| --- | --- |
+| Hard contraindication examples are represented | [app/knowledge/ddi_knowledge.py](polyguard-rl/app/knowledge/ddi_knowledge.py) |
+| Safer alternatives are explicit | [app/knowledge/substitution_rules.py](polyguard-rl/app/knowledge/substitution_rules.py) |
+| Unsafe substitutions and dose escalations are blocked before state mutation | [app/env/verifier.py](polyguard-rl/app/env/verifier.py) |
+| Reward hacking and loop-like behavior are surfaced | [app/env/anti_cheat.py](polyguard-rl/app/env/anti_cheat.py), [docs/reward_design.md](polyguard-rl/docs/reward_design.md) |
+| Baseline failure is traceable by seed and candidate | [basic_llm_vs_polyguard_report.json](polyguard-rl/docs/results/final_submission_evidence/reports/basic_llm_vs_polyguard_report.json), [action_traces.jsonl](polyguard-rl/docs/results/final_submission_evidence/reports/action_traces.jsonl) |
+| Final claims are separated from older smoke artifacts | [final_submission_evidence/README.md](polyguard-rl/docs/results/final_submission_evidence/README.md) |
+
+## Project Map
+
+The implementation lives under [polyguard-rl/](polyguard-rl/).
+
+| Area | Key paths |
+| --- | --- |
+| OpenEnv runtime | [openenv.yaml](polyguard-rl/openenv.yaml), [app/env/env_core.py](polyguard-rl/app/env/env_core.py), [app/env/fastapi_app.py](polyguard-rl/app/env/fastapi_app.py), [server/app.py](polyguard-rl/server/app.py) |
+| Action/state contracts | [app/common/types.py](polyguard-rl/app/common/types.py), [app/common/enums.py](polyguard-rl/app/common/enums.py) |
+| Candidate generation and verifier | [app/models/policy/candidate_builder.py](polyguard-rl/app/models/policy/candidate_builder.py), [app/env/verifier.py](polyguard-rl/app/env/verifier.py) |
+| Reward and anti-cheat | [app/env/reward_router.py](polyguard-rl/app/env/reward_router.py), [app/env/reward_scaling.py](polyguard-rl/app/env/reward_scaling.py), [app/env/anti_cheat.py](polyguard-rl/app/env/anti_cheat.py), [configs/rewards.yaml](polyguard-rl/configs/rewards.yaml) |
+| Multi-agent policy | [app/agents/](polyguard-rl/app/agents/), [docs/agents.md](polyguard-rl/docs/agents.md) |
+| Bandits and baselines | [app/models/baselines/contextual_bandit.py](polyguard-rl/app/models/baselines/contextual_bandit.py), [app/models/baselines/contextual_bandit_policy.py](polyguard-rl/app/models/baselines/contextual_bandit_policy.py), [app/models/baselines/](polyguard-rl/app/models/baselines/) |
+| Training | [app/training/](polyguard-rl/app/training/), [scripts/train_sft_trl.py](polyguard-rl/scripts/train_sft_trl.py), [scripts/train_grpo_trl.py](polyguard-rl/scripts/train_grpo_trl.py), [docs/training.md](polyguard-rl/docs/training.md) |
+| Data | [data/raw/knowledge/drug_knowledge.json](polyguard-rl/data/raw/knowledge/drug_knowledge.json), [data/scenarios/](polyguard-rl/data/scenarios/), [docs/datasets.md](polyguard-rl/docs/datasets.md) |
+| Evaluation | [app/evaluation/](polyguard-rl/app/evaluation/), [scripts/evaluate_all.py](polyguard-rl/scripts/evaluate_all.py), [docs/evaluation.md](polyguard-rl/docs/evaluation.md) |
+| Product API/UI | [app/api/](polyguard-rl/app/api/), [app/ui/frontend/](polyguard-rl/app/ui/frontend/), [docs/ui.md](polyguard-rl/docs/ui.md) |
+| Math | [docs/math.md](polyguard-rl/docs/math.md), [docs/mathematics.md](polyguard-rl/docs/mathematics.md) |
+| Results | [docs/results/final_submission_evidence/](polyguard-rl/docs/results/final_submission_evidence/) |
+
+Supporting docs include [architecture.md](polyguard-rl/docs/architecture.md),
+[environment_design.md](polyguard-rl/docs/environment_design.md),
+[reward_design.md](polyguard-rl/docs/reward_design.md),
+[safety.md](polyguard-rl/docs/safety.md),
+[precision_dosing.md](polyguard-rl/docs/precision_dosing.md),
+[graph_models.md](polyguard-rl/docs/graph_models.md),
+[ablations.md](polyguard-rl/docs/ablations.md),
+[api.md](polyguard-rl/docs/api.md),
+[deployment.md](polyguard-rl/docs/deployment.md),
+[ui.md](polyguard-rl/docs/ui.md),
+[DEMO_RECORDING_SCRIPT.md](polyguard-rl/docs/DEMO_RECORDING_SCRIPT.md), and
+[submission_artifacts.md](polyguard-rl/docs/submission_artifacts.md).
+
+## The OpenEnv Environment
+
+At the center is `PolyGuardEnv`, implemented in
+[app/env/env_core.py](polyguard-rl/app/env/env_core.py). It follows the
+OpenEnv/Gym shape:
+
+```text
+reset(seed, difficulty, sub_environment, scenario_id, patient_id)
+  -> PolyGuardObservation
+
+step(PolyGuardAction)
+  -> (PolyGuardObservation, reward, done, info)
+```
+
+At reset, the environment loads or generates a patient scenario, selects a
+difficulty and sub-environment, computes a risk summary, builds candidate
+actions, estimates uncertainty, and emits a strict observation. At step time,
+the environment parses the action, checks legality, evaluates anti-cheat rules,
+mutates state only if the action is safe, computes decomposed reward, appends a
+trace, and returns detailed `info` fields such as failure reasons, transition
+delta, primary reward channels, invalid-action count, and timeout checks.
+
+![Runtime step flow](polyguard-rl/docs/assets/diagrams/runtime_step_flow.png)
+
+PolyGuard is not one task. It cycles through specialized sub-environments:
+
+| Sub-environment | What it stresses |
+| --- | --- |
+| `DDI` | High-risk drug-drug interaction recognition and resolution |
+| `BANDIT_MINING` | Candidate exploration and shortlist/ranking behavior inspired by bandit search |
+| `REGIMEN_RISK` | General medication burden and regimen optimization |
+| `PRECISION_DOSING` | Dose-hold, dose reduction, renal/hepatic guardrails, monitoring decisions |
+| `LONGITUDINAL_DEPRESCRIBING` | Multi-step taper/deprescribing behavior over a longer horizon |
+| `WEB_SEARCH_MISSING_DATA` | Evidence fetch or review when critical data is missing |
+| `ALTERNATIVE_SUGGESTION` | Safe alternatives and within-class substitution |
+| `NEW_DRUG_DECOMPOSITION` | First-pass reasoning over an unknown or combination medication |
+
+The curriculum in [app/env/curriculum.py](polyguard-rl/app/env/curriculum.py)
+starts with short easy DDI/regimen-risk episodes, then adds bandit and
+alternative-selection tasks, and finally hard cases with precision dosing,
+longitudinal deprescribing, missing data, and new-drug decomposition.
+
+### State and Observation
+
+The latent state is represented by `PolyGuardState` and includes patient
+demographics, active decision mode, step budget, medications, dose buckets,
+comorbidities, labs, vitals, frailty, adherence, monitoring gaps, prior adverse
+event history, burden score, severe-pair count, precision dosing flags,
+unresolved conflicts, action history, cumulative reward, and done state.
+
+The agent does not get every simulator internal. It receives a controlled
+`PolyGuardObservation` with a patient summary, medication table, comorbidity
+summary, organ function, labs/vitals, graph safety summary, burden summary,
+precision dosing flags, unresolved conflicts, candidate actions, step budget,
+action history, warnings, abstention indicators, seed, scenario, difficulty,
+and sub-environment.
+
+This split matters. PolyGuard is a partially observable environment. Missing
+labs and unresolved conflicts are visible as uncertainty signals, not as hidden
+reward traps.
+
+## Action Space and Safety Constraints
+
+PolyGuard deliberately avoids unconstrained text actions. The policy chooses a
+strict `PolyGuardAction` with fields such as `mode`, `action_type`,
+`target_drug`, `replacement_drug`, `dose_bucket`, `taper_days`,
+`monitoring_plan`, `evidence_query`, `new_drug_name`, `candidate_components`,
+`candidate_id`, `confidence`, and `rationale_brief`.
+
+The action types are compact:
+
+| Family | Action types |
+| --- | --- |
+| Regimen | `KEEP_REGIMEN`, `STOP_DRUG`, `SUBSTITUTE_WITHIN_CLASS`, `RECOMMEND_ALTERNATIVE` |
+| Dosing | `REDUCE_DOSE_BUCKET`, `INCREASE_DOSE_BUCKET`, `DOSE_HOLD`, `ORDER_MONITORING_AND_WAIT` |
+| Deprescribing | `TAPER_INITIATE`, `TAPER_CONTINUE` |
+| Evidence and uncertainty | `FETCH_EXTERNAL_EVIDENCE`, `DECOMPOSE_NEW_DRUG`, `REQUEST_SPECIALIST_REVIEW`, `REQUEST_PHARMACIST_REVIEW` |
+
+The candidate builder in
+[app/models/policy/candidate_builder.py](polyguard-rl/app/models/policy/candidate_builder.py)
+generates a bounded candidate set:
+
+```text
+3 <= |C_t| <= 10
+```
+
+Each candidate carries estimated safety delta, burden delta, disease stability,
+uncertainty score, rationale tags, required monitoring, and a legality
+precheck. Policy selection is candidate selection:
+
+```text
+a_t = to_action(c_t), where c_t is in C_t
+```
+
+The verifier in [app/env/verifier.py](polyguard-rl/app/env/verifier.py)
+enforces hard safety constraints before state mutation. It checks that target
+drugs exist when required, substitutions and alternatives are allowed, evidence
+domains are allowlisted, new-drug decomposition includes required components,
+taper-required drugs are not stopped abruptly, renal/hepatic unsafe dose
+escalation is blocked, duplicate therapy and contraindicated replacement pairs
+are blocked, and monitoring/hold actions include a monitoring plan.
+
+Illegal actions can receive reward penalties and become visible in traces, but
+they do not mutate patient state.
+
+## Multi-Agent Policy Stack
+
+The "agents" in PolyGuard are an auditable policy factorization rather than
+free-form independent chatbots. A step flows through:
+
+```text
+MedRec -> Evidence -> GraphSafety -> Dosing -> Candidate
+       -> Supervisor -> Planner -> Critic -> Env -> Explainer
+```
+
+![Multi-agent orchestration](polyguard-rl/docs/assets/diagrams/multi_agent_orchestration.png)
+
+| Agent/module | Role |
+| --- | --- |
+| `MedRecAgent` | Summarizes current regimen and medication burden |
+| `EvidenceAgent` | Retrieves local or fallback evidence when missing data is present |
+| `GraphSafetyAgent` | Scores risky pairs, side-effect load, duplicate therapy, and graph safety patterns |
+| `DosingAgent` | Detects dose-sensitive cases and dose-hold opportunities |
+| `CandidateAgent` | Exposes legal candidate actions from the environment candidate builder |
+| `SupervisorAgent` | Routes to regimen optimization, dose optimization, or review mode |
+| `PlannerAgent` | Selects an action from candidates through the policy provider |
+| `CriticAgent` | Vetoes illegal or unsafe proposed actions and can force review fallback |
+| `ExplainerAgent` | Records grounded rationale for demo, replay, and audit |
+
+The orchestration modes are `sequential_pipeline`, `supervisor_routed`,
+`replan_on_veto`, and `lightweight_debate`. Policy-stack ablations compare
+`bandit-only`, `llm-only`, and `llm+bandit`.
+
+## Contextual Bandits
+
+PolyGuard uses contextual bandits as an inspectable candidate-reranking layer.
+This is where the project most directly echoes the arXiv bandit inspiration:
+unsafe polypharmacy search is combinatorial, so the system should learn which
+regions of the candidate/action space are worth exploring rather than enumerate
+everything.
+
+Each candidate becomes an 8-dimensional feature vector:
+
+```text
+x(c) = [
+  1,
+  I[legality_precheck],
+  estimated_safety_delta,
+  burden_delta,
+  disease_stability_estimate,
+  1 - uncertainty_score,
+  I[mode = DOSE_OPT],
+  I[mode = REVIEW]
+]
+```
+
+An arm is keyed by macro mode and action type:
+
+```text
+arm(c) = mode(c) || ":" || action_type(c)
+```
+
+The LinUCB variant maintains, for each arm `a`:
+
+```text
+A_a = I + sum x x^T
+b_a = sum r x
+theta_a = A_a^{-1} b_a
+
+score_a(x) = theta_a^T x + alpha * sqrt(x^T A_a^{-1} x)
+```
+
+There is also a Thompson-style variant:
+
+```text
+score_a(x) = theta_a^T x + Normal(0, alpha)
+```
+
+This layer can shortlist candidates before the planner emits the final action.
+It is deliberately kept inside the candidate space: the bandit can improve
+ordering and exploration, but it cannot invent an unsafe action outside the
+environment contract.
+
+## Reward Model
+
+The reward model is decomposed on purpose. A single scalar reward is needed for
+RL, but safety-critical RL needs more than one opaque number. PolyGuard logs 13
+component columns and four primary channels on every step.
+
+![Reward decomposition](polyguard-rl/docs/assets/diagrams/reward_decomposition.png)
+
+All reward values are clamped and quantized:
+
+```text
+q(x) = round(clip(x, 0.001, 0.999), 3)
+```
+
+The 13 reward components are:
+
+| Component | Weight | Meaning |
+| --- | ---: | --- |
+| `format_compliance_score` | 0.08 | Action payload conforms to the schema |
+| `candidate_alignment_score` | 0.08 | The model selected a valid candidate-style id |
+| `legality_score` | 0.12 | The verifier accepted the action |
+| `safety_delta_score` | 0.15 | Severe-pair and burden risk decreased |
+| `burden_improvement_score` | 0.08 | Dose-weighted medication burden improved |
+| `disease_stability_score` | 0.10 | The action did not destabilize underlying disease management |
+| `dosing_quality_score` | 0.08 | Dose-sensitive routing/action quality |
+| `abstention_quality_score` | 0.06 | Review/abstention is appropriate under uncertainty |
+| `efficiency_score` | 0.06 | The action uses the finite step budget well |
+| `process_fidelity_score` | 0.06 | The action follows task-specific process expectations |
+| `explanation_grounding_score` | 0.03 | The rationale is present and grounded |
+| `anti_cheat_score` | 0.06 | Reward-hacking checks did not fire |
+| `uncertainty_calibration_score` | 0.04 | Confidence matches observable uncertainty |
+
+The scalar reward is a weighted average:
+
+```text
+R_env(s_t, a_t, s_{t+1}) = q( sum_i w_i c_i / sum_i w_i )
+```
+
+Safety-heavy terms dominate the total weight:
+
+```text
+legality + safety_delta + burden + disease_stability + anti_cheat
+= 0.12 + 0.15 + 0.08 + 0.10 + 0.06
+= 0.51
+```
+
+The four primary reward channels are:
+
+| Channel | Component family |
+| --- | --- |
+| `safety_legality` | legality, candidate alignment, anti-cheat, uncertainty calibration |
+| `clinical_improvement` | safety delta, burden improvement, disease stability |
+| `dosing_quality` | dosing quality and abstention quality |
+| `process_integrity` | format compliance, efficiency, process fidelity, explanation grounding |
+
+These channels are emitted in `info.primary_reward_channels`, GRPO reward logs,
+reports, plots, and ablation summaries.
+
+## Anti-Cheat and Failure Visibility
+
+RL policies exploit reward functions. PolyGuard makes common shortcut failures
+explicit: repeated action loops, excessive keep-regimen behavior, excessive
+review/abstention behavior, candidate ID mismatch, candidate outside the legal
+set, hidden high-risk DDI no-op behavior, parser exploit patterns in rationales,
+and retries of failed no-op actions.
+
+If an exploit is detected:
+
+```text
+anti_cheat_score = 0.001
+done = true
+termination_reason = "exploit_detection"
+```
+
+Episodes can also terminate on step budget exhaustion, repeated invalid
+actions, safety-veto threshold, patient destabilization, safe resolution,
+wall-clock timeout, or per-step timeout.
+
+![Episode state machine](polyguard-rl/docs/assets/diagrams/episode_state_machine.png)
+
+## Mathematics
+
+PolyGuard can be read as a finite-horizon constrained partially observable
+Markov decision process:
+
+```text
+M = (S, A, O, T, R, H, C)
+```
+
+where `S` is latent patient/regimen state, `A` is the constrained medication
+action set, `O` is the controlled observation, `T(s' | s, a)` is the transition
+function, `R(s, a, s')` is verifier-backed reward, `H` is the episode horizon,
+and `C(s, a)` is the hard safety/legality constraint predicate.
+
+The objective is:
+
+```text
+maximize_pi   E_pi [ sum_{t=0}^{H-1} R(s_t, a_t, s_{t+1}) ]
+subject to    C(s_t, a_t) = 1 whenever possible
+```
+
+There is no explicit discount factor in the runtime. Time preference enters
+through finite horizons and the efficiency reward:
+
+```text
+efficiency_t = q(1 - step_count_t / (max_steps + 1))
+```
+
+State transition is two-gated:
+
+```text
+if verifier(s_t, a_t).legal and not anti_cheat(s_t, a_t):
+    s_{t+1} = T(s_t, a_t)
+else:
+    s_{t+1} = rollback_state_with_failed_action_record(s_t, a_t)
+```
+
+Risk-like deltas become reward through:
+
+```text
+delta_reward(pre, post) = q(0.5 + 0.6 * (pre - post))
+```
+
+For burden and contraindicated-pair improvement:
+
+```text
+burden_reward = delta_reward(pre_burden, post_burden)
+pair_reward   = delta_reward(pre_pairs, post_pairs)
+
+safety_delta_score =
+  q(0.65 * pair_reward + 0.35 * burden_reward) if legal
+  0.001 otherwise
+```
+
+GRPO uses environment execution as the reward function. For each prompt, the
+model emits candidate completions; PolyGuard parses the candidate id, resets a
+deterministic environment using the recorded seed and scenario fields, executes
+one step, and returns reward. The training reward combines environment reward
+with a legality bonus:
+
+```text
+legal_bonus = 0.95 if action is legal else 0.05
+
+R_GRPO = q(0.80 * R_env + 0.20 * legal_bonus)
+```
+
+Conceptually, GRPO forms a within-prompt advantage:
+
+```text
+A_i = (R_i - mean_j R_j) / (std_j R_j + epsilon)
+```
+
+and optimizes a clipped policy-ratio objective with KL regularization. The
+optimizer mechanics are TRL's; PolyGuard's contribution is the verifier-backed
+reward function and the controlled action/state environment. The expanded
+derivation is in
+[polyguard-rl/docs/mathematics.md](polyguard-rl/docs/mathematics.md).
+
+## Data and Dataset Pipeline
+
+The data pipeline builds a compact medication-safety substrate from local drug
+knowledge, synthetic patients, scenario files, retrieval text, and optional
+external augmentation.
+
+![Data and training pipeline](polyguard-rl/docs/assets/diagrams/data_training_pipeline.png)
+
+The dataset design is documented in
+[docs/datasets.md](polyguard-rl/docs/datasets.md). The local generated
+pipeline produces these processed artifacts and counts:
+
+| Artifact | Count | Path |
+| --- | ---: | --- |
+| Normalized drug rows | 10 | `data/processed/normalized_drugs.parquet` |
+| Drug class rows | 10 | `data/processed/drug_classes.parquet` |
+| Interaction rows | 2 | `data/processed/interactions.parquet` |
+| Graph edges | 18 | `data/processed/graph_edges.parquet` |
+| Synthetic patients | 20 | `data/processed/patients_synthetic.parquet` |
+| Retrieval documents | 8 | `data/processed/retrieval_corpus.jsonl` |
+| Easy scenarios | 100 | [data/scenarios/easy/](polyguard-rl/data/scenarios/easy/) |
+| Medium scenarios | 200 | [data/scenarios/medium/](polyguard-rl/data/scenarios/medium/) |
+| Hard scenarios | 200 | [data/scenarios/hard/](polyguard-rl/data/scenarios/hard/) |
+| Local small SFT rows | 80 | `data/processed/training_corpus_sft.jsonl` |
+| Local small GRPO prompts | 80 | `data/processed/training_corpus_grpo_prompts.jsonl` |
+
+The provenance manifest generated by the local pipeline records source policy
+and counts at `data/processed/provenance_manifest.json`.
+
+Additional data-governance and rule artifacts are produced or consumed by the
+pipeline:
+
+| Artifact | Why it matters |
+| --- | --- |
+| `data/processed/ingested_sources.json` | Source ingestion ledger used by the local build |
+| `data/processed/feature_dictionary.json` | Names and meanings of structured model features |
+| `data/processed/burden_rules.yaml` | Medication-burden and duplicate-therapy rules |
+| `data/processed/substitution_rules.yaml` | Data-level safer-substitution rules |
+| `data/processed/taper_rules.yaml` | Deprescribing and taper requirements |
+| `data/retrieval_index/index.json` | Retrieval index over local evidence chunks |
+
+The local knowledge seed is
+[data/raw/knowledge/drug_knowledge.json](polyguard-rl/data/raw/knowledge/drug_knowledge.json).
+It contains drug classes, example high-risk pairs, renal and hepatic flags,
+side-effect tags, substitution rules, and taper requirements. The processed
+tables then feed graph modeling, candidate generation, environment scenarios,
+retrieval, SFT rows, and GRPO prompts.
+
+The full training/evidence runs used 2,000 examples per Qwen model, recorded in
+the final reports under
+[docs/results/final_submission_evidence/reports/](polyguard-rl/docs/results/final_submission_evidence/reports/).
+
+## Models Inside the Environment
+
+PolyGuard combines learned and rule-backed components:
+
+- Graph safety model:
+  [app/models/graph/](polyguard-rl/app/models/graph/) produces regimen
+  embeddings, pairwise DDI severity, severe-alert probability, and side-effect
+  tag probabilities.
+- Tabular risk model:
+  [app/models/tabular/](polyguard-rl/app/models/tabular/) supports calibrated
+  patient/regimen risk heads and evaluation.
+- Dosing model:
+  [app/models/dosing/](polyguard-rl/app/models/dosing/) models dose-sensitive
+  states with target attainment, toxicity, underdose risk, organ stress,
+  interaction load, and monitoring need.
+- Retrieval:
+  [app/models/retrieval/](polyguard-rl/app/models/retrieval/) and
+  [app/knowledge/](polyguard-rl/app/knowledge/) provide local evidence chunks,
+  drug rules, renal/hepatic guardrails, duplicate therapy rules, substitution
+  rules, taper rules, burden scoring, and side-effect ontology.
+- Active model runtime:
+  [app/models/policy/active_model.py](polyguard-rl/app/models/policy/active_model.py)
+  discovers activated artifacts from `checkpoints/active/active_model_manifest.json`;
+  the tracked evidence mirror includes
+  [docs/results/active_model_manifest.json](polyguard-rl/docs/results/active_model_manifest.json).
+  The provider load order prefers a GRPO adapter, then merged model, then SFT
+  adapter.
+- Provider runtime:
+  [app/models/policy/provider_runtime.py](polyguard-rl/app/models/policy/provider_runtime.py)
+  is Transformers-first, with optional Ollama when enabled. If model loading is
+  unavailable, the runtime falls back to deterministic safety ranking.
+
+Tracked support-model reports show that the environment is not only an LLM
+wrapper:
+
+| Component | Report | Current tracked result |
+| --- | --- | --- |
+| Graph model | [docs/results/graph_train.json](polyguard-rl/docs/results/graph_train.json) | `status: trained`, `num_samples: 180`, artifact path `outputs/models/graph_model.pkl` |
+| Tabular risk model | [docs/results/risk_train.json](polyguard-rl/docs/results/risk_train.json) | `status: trained`, `dataset_size: 180`, `train_mae: 0.0033`, artifact path `outputs/models/tabular_risk.pkl` |
+| Dose surrogate model | [docs/results/dose_train.json](polyguard-rl/docs/results/dose_train.json) | `status: trained`, `dataset_size: 120`, `train_mae: 0.0025`, artifact path `outputs/models/dose_model.pkl` |
+
+The hard-coded contraindicated seed pairs in
+[app/knowledge/ddi_knowledge.py](polyguard-rl/app/knowledge/ddi_knowledge.py)
+include `warfarin_like` + `nsaid_like` and `benzodiazepine_like` +
+`opioid_like`. Substitution rules in
+[app/knowledge/substitution_rules.py](polyguard-rl/app/knowledge/substitution_rules.py)
+include safer alternatives such as `nsaid_like -> acetaminophen_like`,
+`nsaid_like -> topical_nsaid_like`, `benzodiazepine_like ->
+non_benzo_sleep_support`, and `opioid_like -> non_opioid_analgesic`.
+
+### Precision Dosing
+
+Precision dosing uses sensitive classes such as anticoagulants, sedatives, and
+glucose-lowering drugs. The dosing agent and surrogate model are implemented in
+[app/agents/dosing_agent.py](polyguard-rl/app/agents/dosing_agent.py) and
+[app/models/dosing/](polyguard-rl/app/models/dosing/).
+
+The surrogate PK/PD transition in
+[app/models/dosing/surrogate_pkpd.py](polyguard-rl/app/models/dosing/surrogate_pkpd.py)
+uses effect, toxicity, underdose, organ stress, and interaction load:
+
+```text
+effective_delta = dose_delta * (1 - min(0.6, organ_factor * 0.4))
+
+effect' =
+  clip(effect + 0.28 * effective_delta - 0.05 * interaction_factor, 0, 1)
+
+toxicity_gain =
+  max(0, dose_delta) * (0.35 + 0.25 * organ_factor + 0.20 * interaction_factor)
+
+toxicity' =
+  clip(0.85 * toxicity + toxicity_gain, 0, 1)
+
+underdose' =
+  clip(1 - effect' + 0.15 * max(0, -dose_delta), 0, 1)
+```
+
+The higher-level dosing metrics use target attainment, toxicity avoidance,
+underdose risk, and monitoring need:
+
+```text
+target_attainment = 1 - abs(effect_level - 0.62)
+toxicity_proxy    = toxicity_level + 0.20 * organ_stress + 0.12 * interaction_load
+measurement_need  = max(toxicity_proxy, underdose_proxy)
+```
+
+## Training and Post-Training
+
+The training stack is deliberately staged:
+
+1. Build structured data, scenarios, retrieval records, SFT examples, and GRPO
+   prompts.
+2. Run SFT with TRL to teach the model the candidate-id format and obvious
+   clinical priors.
+3. Run GRPO with environment-backed reward, where sampled candidate completions
+   are executed in PolyGuardEnv and scored by the verifier/reward router.
+4. Track sampled generations, reward components, primary reward channels,
+   legality, anti-cheat events, and training curves.
+5. Run policy-stack ablations and baseline comparisons.
+6. Merge or export adapters safely.
+7. Validate post-save inference from the saved artifact, not from an in-memory
+   training object.
+8. Generate reports, charts, action traces, and final artifact manifests.
+
+The relevant training source files are
+[scripts/train_sft_trl.py](polyguard-rl/scripts/train_sft_trl.py),
+[scripts/train_grpo_trl.py](polyguard-rl/scripts/train_grpo_trl.py),
+[app/training/sft_trl.py](polyguard-rl/app/training/sft_trl.py),
+[app/training/grpo_trl.py](polyguard-rl/app/training/grpo_trl.py),
+[app/training/reward_functions.py](polyguard-rl/app/training/reward_functions.py),
+[app/training/openenv_wrapper.py](polyguard-rl/app/training/openenv_wrapper.py),
+and [app/hf_space/training_runner.py](polyguard-rl/app/hf_space/training_runner.py).
+
+The one-run notebook is
+[polyguard-rl/PolyGuard_SFT_GRPO_One_Run_Runner.ipynb](polyguard-rl/PolyGuard_SFT_GRPO_One_Run_Runner.ipynb).
+It is the accessible Colab/HF workflow for building data, running checks,
+launching training, pulling reports, generating charts, validating inference,
+activating a model, deploying the product Space, and running acceptance checks.
+
+The modular notebook series is:
+
+- [01_data_building.ipynb](polyguard-rl/notebooks/01_data_building.ipynb)
+- [02_knowledge_graph.ipynb](polyguard-rl/notebooks/02_knowledge_graph.ipynb)
+- [03_risk_models.ipynb](polyguard-rl/notebooks/03_risk_models.ipynb)
+- [04_environment_validation.ipynb](polyguard-rl/notebooks/04_environment_validation.ipynb)
+- [05_sft_debug.ipynb](polyguard-rl/notebooks/05_sft_debug.ipynb)
+- [06_grpo_debug.ipynb](polyguard-rl/notebooks/06_grpo_debug.ipynb)
+- [07_policy_analysis.ipynb](polyguard-rl/notebooks/07_policy_analysis.ipynb)
+- [08_dosing_analysis.ipynb](polyguard-rl/notebooks/08_dosing_analysis.ipynb)
+- [09_training_loop.ipynb](polyguard-rl/notebooks/09_training_loop.ipynb)
+
+For exact local and remote execution details, use
+[docs/training.md](polyguard-rl/docs/training.md) and
+[docs/submission_artifacts.md](polyguard-rl/docs/submission_artifacts.md).
+This blog focuses on architecture, data, evaluation, and evidence rather than
+private or environment-specific commands.
+
+## Training Curves and Model Results
+
+The final curated evidence lives in
+[polyguard-rl/docs/results/final_submission_evidence/](polyguard-rl/docs/results/final_submission_evidence/).
+It replaces earlier smoke-run charts and older 0.5B/1.5B-only views.
+
+### SFT Loss Across Qwen Runs
+
+![SFT loss curves across Qwen runs](polyguard-rl/docs/results/final_submission_evidence/charts/curated/training/sft_loss_curves_all_models.png)
+
+The SFT curves, post-save valid rates, and token-accuracy histories show that
+the models learned the candidate-id output contract rather than only producing
+unconstrained prose. The visible curves drop from roughly `3.0-3.6` initial
+loss to low final loss across all three Qwen sizes.
+
+![Qwen 3B SFT training loss](polyguard-rl/docs/results/final_submission_evidence/charts/curated/training/qwen_3b_sft_training_loss.png)
+
+The tracked per-model summaries are:
+
+| Run | Model | Epochs | Final step | Runtime | Key SFT metrics |
+| --- | --- | ---: | ---: | ---: | --- |
+| `qwen-qwen2-5-0-5b-instruct` | `Qwen/Qwen2.5-0.5B-Instruct` | 2 | 2,000 | `234.6302s` | loss `3.0856 -> 0.0626`, best `0.0057`, train loss `0.1923`, token accuracy `0.9717`, valid rate `1.0`, avg env reward `0.726`, latency `1.839s` |
+| `qwen-qwen2-5-1-5b-instruct` | `Qwen/Qwen2.5-1.5B-Instruct` | 2 | 4,000 | `483.7085s` | loss `2.9686 -> 0.0681`, best `0.0009`, train loss `0.1152`, token accuracy `0.9726`, valid rate `1.0`, avg env reward `0.726`, latency `2.158s` |
+| `qwen-qwen2-5-3b-instruct` | `Qwen/Qwen2.5-3B-Instruct` | 2 | 2,000 | `715.2908s` | loss `3.5687 -> 0.054`, best `0.0022`, train loss `0.1569`, token accuracy `0.9750`, SFT avg env reward `0.781`, SFT latency `2.863s` |
+
+Each SFT run used 2,000 examples. The 0.5B and 3B runs recorded 2,001 history
+rows including the final trainer summary; the 1.5B run recorded 4,001 history
+rows because its batch configuration produced 4,000 final steps.
+
+### GRPO Reward Curve
+
+![Qwen 3B GRPO reward curve](polyguard-rl/docs/results/final_submission_evidence/charts/curated/training/qwen_3b_grpo_reward_curve.png)
+
+![Qwen 3B GRPO training loss](polyguard-rl/docs/results/final_submission_evidence/charts/curated/training/qwen_3b_grpo_loss_curve.png)
+
+The complete GRPO evidence is available for Qwen 3B:
+
+- Backend: `trl_transformers`
+- Model: `Qwen/Qwen2.5-3B-Instruct`
+- Records: `2000`
+- Epochs: `1.0`
+- Final step: `2000`
+- Runtime: `6873.9375s` (`1.91h`)
+- Reward samples: `4000`
+- GRPO average reward: `0.767`
+- GRPO reward history: min `0.376`, max `0.880`, last `0.812`, average `0.76685`
+- GRPO train loss: `0.000002665`
+- Post-save GRPO valid rate: `1.0`
+- Post-save GRPO average environment reward: `0.726`
+- Post-save GRPO average latency: `3.681s`
+- Artifact path recorded in the report: `checkpoints/sweeps/qwen-qwen2-5-3b-instruct/grpo_adapter`
+
+Source reports:
+[grpo_trl_run.json](polyguard-rl/docs/results/final_submission_evidence/reports/grpo_trl_run.json),
+[postsave_inference_grpo.json](polyguard-rl/docs/results/final_submission_evidence/reports/postsave_inference_grpo.json),
+and
+[submission_summary.json](polyguard-rl/docs/results/final_submission_evidence/reports/submission_summary.json).
+
+### SFT vs GRPO by Model
+
+![SFT vs GRPO verifier reward by model](polyguard-rl/docs/results/final_submission_evidence/charts/curated/model_comparison/sft_vs_grpo_reward_by_model.png)
+
+This chart is intentionally transparent about artifact availability. Qwen 0.5B
+and 1.5B have SFT reports/histories and post-save SFT evidence in the repo, but
+their adapter directories were not present in the local/final artifact mirrors
+at packaging time. Qwen 3B has the complete SFT plus GRPO artifact set.
+
+The packaged manifest records Qwen 3B as complete with 125 checkpoint files
+(`433,208,536` bytes), 11 SFT adapter files (`30,655,905` bytes), 11 GRPO
+adapter files (`30,656,841` bytes), and 9 report files (`5,930,214` bytes).
+Qwen 0.5B and 1.5B are retained as report/post-save evidence only.
+
+Manifest:
+[docs/results/final_submission_evidence/manifest.json](polyguard-rl/docs/results/final_submission_evidence/manifest.json).
+
+### Product Pipeline vs Basic LLM Proxy
+
+![Basic LLM vs full PolyGuard pipeline](polyguard-rl/docs/results/final_submission_evidence/charts/curated/product_over_basic_llm/basic_llm_vs_full_pipeline_reward.png)
+
+Matched-seed evaluation compares a basic LLM-style first-legal proxy, an
+SFT-style safety ranker, and the full PolyGuard orchestrated pipeline. The same
+PolyGuard verifier/reward system judges all three.
+
+| Policy | Episodes | Avg reward | Legality rate | Failure/exploit rate | Candidate diversity |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| Basic LLM proxy | 8 | `0.762` | `1.0` | `0.25` | 1 |
+| SFT policy proxy | 8 | `0.818` | `1.0` | `0.0` | 2 |
+| Full PolyGuard pipeline | 8 | `0.805` | `1.0` | `0.0` | 2 |
+
+The full pipeline improves average verifier reward over the basic LLM proxy by
+`+0.043` while reducing visible failure/exploit rate from `0.25` to `0.0`.
+
+![Reward delta by matched seed](polyguard-rl/docs/results/final_submission_evidence/charts/curated/product_over_basic_llm/reward_delta_by_seed.png)
+
+Two matched seeds expose the core failure mode: the basic policy repeatedly
+kept a regimen despite the hidden `warfarin_like` + `nsaid_like` DDI holdout,
+triggering `holdout_ddi_not_addressed`. The full pipeline selected safer dose
+or hold candidates and avoided those failure reasons.
+
+Source:
+[basic_llm_vs_polyguard_report.json](polyguard-rl/docs/results/final_submission_evidence/reports/basic_llm_vs_polyguard_report.json).
+
+### Reward Components and Channels
+
+![Reward component bars](polyguard-rl/docs/results/final_submission_evidence/charts/curated/reward_and_safety/reward_component_bars.png)
+
+![Primary reward channel bars](polyguard-rl/docs/results/final_submission_evidence/charts/curated/reward_and_safety/primary_reward_channel_bars.png)
+
+The reward charts are as important as the scalar reward curve. They show
+whether the model is improving by becoming safer and more process-faithful or
+merely exploiting one easy component. The reports log the full 13-component
+reward vector and the four primary channels for GRPO and evaluation runs.
+
+For Qwen 3B GRPO, the tracked average primary channels are:
+
+| Channel | Average |
+| --- | ---: |
+| `safety_legality` | `0.816` |
+| `clinical_improvement` | `0.609` |
+| `dosing_quality` | `0.543` |
+| `process_integrity` | `0.875` |
+
+### Post-Save Inference
+
+![Inference validity and reward](polyguard-rl/docs/results/final_submission_evidence/charts/curated/inference/inference_validity_reward.png)
+
+Post-save inference is separate from training. The exported/activated artifact
+is loaded and asked to choose candidate ids on held prompt samples. The Qwen 3B
+GRPO adapter path produced:
+
+- `model_source: adapter`
+- `samples: 5`
+- `valid_rate: 1.0`
+- `avg_env_reward: 0.726`
+- `avg_latency_seconds: 3.681`
+
+The caveat matters: `valid_rate: 1.0` means the output was parseable and
+executable as a candidate selection. In the five-sample Qwen 3B post-save GRPO
+report, four valid samples still terminated with `exploit_detection`. That is
+retained as safety evidence, because PolyGuard's job is to expose suspicious or
+loop-like behavior instead of hiding it behind a clean parse metric.
+
+## Agentic Evaluation
+
+Evaluation is not one benchmark number. The evaluation stack under
+[app/evaluation/](polyguard-rl/app/evaluation/) includes offline policy
+evaluation, safety evaluation, dosing evaluation, robustness under missing labs
+and noisy inputs, calibration and abstention evaluation, process fidelity,
+subgroup summaries, explainability grounding, baseline comparison, policy
+ablations, failure mining, and action traces.
+
+The tracked benchmark report records:
+
+| Metric family | Result |
+| --- | --- |
+| Offline avg reward | `0.772833` |
+| Offline legal rate | `1.0` |
+| Severe violation rate | `0.0` |
+| Illegal step rate | `0.0` |
+| Dosing target attainment | `0.75` |
+| Dosing toxicity avoidance | `1.0` |
+| Missing-labs safety rate | `0.666667` |
+| Noisy-dose, conflicting-meds, alias-noise, hidden-duplicate, wrong-candidate-id, stale-evidence, delayed-ADE safety/resilience | `1.0` |
+| Calibration ECE proxy | `0.08625` |
+| Process fidelity | `0.92` |
+| Explainability grounding | `0.8` |
+
+Source:
+[docs/results/benchmark_report.json](polyguard-rl/docs/results/benchmark_report.json).
+
+The improvement gate compares baseline and candidate reports:
+
+| Gate dimension | Delta |
+| --- | ---: |
+| Average reward | `+0.025833` |
+| Legality rate | `0.0` non-regression |
+| Success rate | `0.0` non-regression |
+| Process fidelity | `+0.92` |
+| Timeout rate | `0.0` non-regression |
+| Failure visibility | `0.0` non-regression |
+
+Source:
+[docs/results/improvement_report.json](polyguard-rl/docs/results/improvement_report.json).
+
+### Policy Ablation Results
+
+| Stack | Avg reward | Legality | Visible failure rate | Exploit detections | Interpretation |
+| --- | ---: | ---: | ---: | ---: | --- |
+| `bandit_only` | `0.779625` | `1.0` | `0.0625` | 2 | Strong deterministic shortlist behavior with low failure visibility |
+| `llm_only` | `0.772391` | `1.0` | `0.3043` | 7 | Legal, but more loop-like failure behavior |
+| `llm+bandit` | `0.764739` | `1.0` | `0.3043` | 7 | Current combined stack needs tighter exploration/control in these ablation settings |
+
+![Policy ablation reward](polyguard-rl/docs/results/final_submission_evidence/charts/curated/policy_ablation/policy_ablation_reward.png)
+
+The point of these ablations is not to claim every combined policy is always
+better. The point is that PolyGuard can localize behavior: legality remains
+high, while failure mining shows whether a stack is looping, over-reviewing,
+or selecting non-improving candidates.
+
+Source:
+[policy_ablation_report.json](polyguard-rl/docs/results/final_submission_evidence/reports/policy_ablation_report.json).
+
+## OpenEnv and Product Surfaces
+
+The OpenEnv package is compact:
+
+```yaml
+spec_version: 1
+name: polyguard-openenv
+runtime: fastapi
+app: app.env.fastapi_app:app
+port: 8100
+```
+
+The OpenEnv runtime exposes `POST /reset`, `POST /step`, `GET /state`,
+`GET /metadata`, `GET /schema`, `POST /mcp`, `GET /health`, `GET /ws`, and
+backward-compatible `/env/*` routes.
+
+The product API in [app/api/routes.py](polyguard-rl/app/api/routes.py) wraps
+the environment, orchestrator, policy runtime, evaluation, evidence search,
+cases, metrics, and medication-alternative tooling. Product-facing endpoints
+include `/env/reset`, `/env/step_candidate`, `/agents/orchestrate`,
+`/policy/infer`, `/policy/model_status`, `/eval/run_policy`,
+`/metrics/training`, `/evidence/query`, and `/tools/medication_alternatives`.
+
+![Deployment topology](polyguard-rl/docs/assets/diagrams/deployment_topology.png)
+
+## Operations and Deployment
+
+The repository keeps deployment and artifact operations explicit:
+
+| Surface | Files |
+| --- | --- |
+| Local/container runtime | [Dockerfile](polyguard-rl/Dockerfile), [Dockerfile.space](polyguard-rl/Dockerfile.space), [docker-compose.yml](polyguard-rl/docker-compose.yml), [requirements.txt](polyguard-rl/requirements.txt), [requirements-space.txt](polyguard-rl/requirements-space.txt) |
+| Product Space/API deployment | [scripts/deploy_space.sh](polyguard-rl/scripts/deploy_space.sh), [scripts/deploy_space_api.py](polyguard-rl/scripts/deploy_space_api.py), [docs/deployment.md](polyguard-rl/docs/deployment.md) |
+| Training and evidence Spaces | [scripts/deploy_training_space.py](polyguard-rl/scripts/deploy_training_space.py), [scripts/monitor_training_space_status.py](polyguard-rl/scripts/monitor_training_space_status.py), [app/hf_space/training_runner.py](polyguard-rl/app/hf_space/training_runner.py), [app/hf_space/evidence_runner.py](polyguard-rl/app/hf_space/evidence_runner.py) |
+| Artifact packaging and activation | [scripts/deploy_final_artifact_space.py](polyguard-rl/scripts/deploy_final_artifact_space.py), [scripts/package_active_model_bundle.py](polyguard-rl/scripts/package_active_model_bundle.py), [scripts/install_hf_active_bundle.py](polyguard-rl/scripts/install_hf_active_bundle.py), [docs/results/active_model_manifest.json](polyguard-rl/docs/results/active_model_manifest.json) |
+| Submission validation | [scripts/acceptance_gate.py](polyguard-rl/scripts/acceptance_gate.py), [scripts/validate_submission_links.py](polyguard-rl/scripts/validate_submission_links.py), [docs/submission_checklist.md](polyguard-rl/docs/submission_checklist.md), [docs/submission_artifacts.md](polyguard-rl/docs/submission_artifacts.md) |
+
+The important operational distinction is that local smoke artifacts, remote
+training-space logs, final artifact packaging, and active-model installation
+are separate stages. Final claims are tied to the curated evidence bundle, not
+to whichever intermediate output directory happens to exist in a checkout.
+
+## The Workbench UI
+
+The UI is a React 18 + Vite + TypeScript workbench under
+[app/ui/frontend/](polyguard-rl/app/ui/frontend/). It is not the environment
+itself; it is an operator surface over the API and OpenEnv runtime.
+
+[Live workbench Space](https://huggingface.co/spaces/TheJackBright/polyguard-openenv-workbench)
+
+![Frontend runtime surface](polyguard-rl/docs/assets/diagrams/frontend_runtime_surface.png)
+
+The main views cover patient workbench, episode replay, policy comparison and
+policy lab, precision dosing, training monitor, safety inspector, candidate
+actions, reward panel, episode trace, and alternative medication search through
+`/tools/medication_alternatives`.
+
+The Patient Workbench shows the active model chip, current scenario, candidate
+set, agent-vs-environment flow, reward breakdown, and action trace without
+requiring the reader to inspect raw JSON. The UI is intentionally a workbench,
+not a polished clinical application.
+
+### UI Sequence
+
+The five UI screenshots are checked in under `polyguard-rl/docs/UI Images/`.
+
+1. The workbench opens with model truth, live episode context, scenario status,
+   candidate count, and reward state.
+
+![PolyGuard workbench overview](polyguard-rl/docs/UI%20Images/1.jpeg)
+
+2. The episode panel makes the patient, task, difficulty, sub-environment, risk
+   delta, and candidate-action console visible without reading raw JSON.
+
+![Episode overview and candidate console](polyguard-rl/docs/UI%20Images/2.jpeg)
+
+3. Candidate selection is paired with reward-channel feedback, current
+   medications, and blocked/available action visibility.
+
+![Candidate actions and reward channels](polyguard-rl/docs/UI%20Images/3.jpeg)
+
+4. After an action, the workbench exposes history, warnings, decision payload,
+   grounded facts, explanation, evidence, and event logs.
+
+![Action history, decision payload, and evidence](polyguard-rl/docs/UI%20Images/4.jpeg)
+
+5. The alternatives tool surfaces medication substitutions from the current
+   regimen and links out to source labels.
+
+![Medication alternatives tool](polyguard-rl/docs/UI%20Images/5.jpeg)
+
+## Demo Videos
+
+### [UI Walkthrough Video](https://drive.google.com/file/d/1YOzad5gvx-tSmGzJNuBgokBF4-dX2T2H/view?usp=sharing)
+
+This walkthrough shows the deployed workbench surface, including the live model
+chip, episode context, candidate actions, reward panels, and evidence-oriented
+patient review flow.
+
+### [Agent In Action: Action Button Demo](https://drive.google.com/file/d/1eHk1v0OYJRrLWVO97ZclN05MYHxmNnmc/view?usp=sharing)
+
+This demo focuses on what the action button does: selecting a candidate,
+submitting it through the environment, producing a verifier-scored transition,
+and exposing the resulting reward, action history, warnings, and explanation.
+
+### [World Model Tool: Tavily and OpenFDA Alternative Suggestions](https://drive.google.com/file/d/1GaUyyaXaBCHjhHFbpkprojNt5pLNAoYi/view?usp=sharing)
+
+This tool demo shows the world-model support path for alternative medication
+suggestions, using Tavily and the OpenFDA government database to retrieve
+candidate alternatives and side-effect evidence for safer review.
+
+## How a Reviewer Should Read the Repository
+
+For a fresh reviewer, the intended path is:
+
+1. Read the artifact index:
+   [polyguard-rl/docs/submission_artifacts.md](polyguard-rl/docs/submission_artifacts.md).
+2. Inspect the final curated evidence:
+   [polyguard-rl/docs/results/final_submission_evidence/README.md](polyguard-rl/docs/results/final_submission_evidence/README.md).
+3. Open the one-run notebook:
+   [PolyGuard_SFT_GRPO_One_Run_Runner.ipynb](polyguard-rl/PolyGuard_SFT_GRPO_One_Run_Runner.ipynb).
+4. For local smoke work, follow [docs/training.md](polyguard-rl/docs/training.md)
+   and the local scripts
+   [scripts/run_env_local.sh](polyguard-rl/scripts/run_env_local.sh),
+   [scripts/run_api_local.sh](polyguard-rl/scripts/run_api_local.sh), and
+   [scripts/run_ui_local.sh](polyguard-rl/scripts/run_ui_local.sh).
+5. For full training/reproduction, use the notebook or training docs rather
+   than copying private artifact commands out of old drafts.
+6. For final public artifacts, use the final artifact Space:
+   [adithya9903/polyguard-openenv-final-artifacts](https://huggingface.co/spaces/adithya9903/polyguard-openenv-final-artifacts).
+
+## Evidence and Artifact Inventory
+
+Important evidence paths:
+
+- Final overview:
+  [docs/results/final_submission_evidence/README.md](polyguard-rl/docs/results/final_submission_evidence/README.md)
+- Artifact manifest:
+  [docs/results/final_submission_evidence/manifest.json](polyguard-rl/docs/results/final_submission_evidence/manifest.json)
+- Three-model summary:
+  [docs/results/final_submission_evidence/reports/submission_summary.json](polyguard-rl/docs/results/final_submission_evidence/reports/submission_summary.json)
+- Qwen 3B GRPO report:
+  [docs/results/final_submission_evidence/reports/grpo_trl_run.json](polyguard-rl/docs/results/final_submission_evidence/reports/grpo_trl_run.json)
+- Post-save GRPO inference:
+  [docs/results/final_submission_evidence/reports/postsave_inference_grpo.json](polyguard-rl/docs/results/final_submission_evidence/reports/postsave_inference_grpo.json)
+- Basic LLM vs PolyGuard:
+  [docs/results/final_submission_evidence/reports/basic_llm_vs_polyguard_report.json](polyguard-rl/docs/results/final_submission_evidence/reports/basic_llm_vs_polyguard_report.json)
+- Policy ablation:
+  [docs/results/final_submission_evidence/reports/policy_ablation_report.json](polyguard-rl/docs/results/final_submission_evidence/reports/policy_ablation_report.json)
+- Action traces:
+  [docs/results/final_submission_evidence/reports/action_traces.jsonl](polyguard-rl/docs/results/final_submission_evidence/reports/action_traces.jsonl)
+- Curated charts:
+  [docs/results/final_submission_evidence/charts/curated/README.md](polyguard-rl/docs/results/final_submission_evidence/charts/curated/README.md)
+
+Important tests:
+
+| Category | Tests |
+| --- | --- |
+| Environment contract | [tests/test_openenv_contract.py](polyguard-rl/tests/test_openenv_contract.py), [tests/test_env_reset.py](polyguard-rl/tests/test_env_reset.py), [tests/test_env_step.py](polyguard-rl/tests/test_env_step.py), [tests/test_env_step_flow.py](polyguard-rl/tests/test_env_step_flow.py), [tests/test_future_subenvs.py](polyguard-rl/tests/test_future_subenvs.py) |
+| Reward and safety | [tests/test_reward_functions.py](polyguard-rl/tests/test_reward_functions.py), [tests/test_reward_range.py](polyguard-rl/tests/test_reward_range.py), [tests/test_reward_channels.py](polyguard-rl/tests/test_reward_channels.py), [tests/test_anti_cheat.py](polyguard-rl/tests/test_anti_cheat.py), [tests/test_constraints.py](polyguard-rl/tests/test_constraints.py), [tests/test_timeout_logic.py](polyguard-rl/tests/test_timeout_logic.py) |
+| Policy and runtime | [tests/test_agents.py](polyguard-rl/tests/test_agents.py), [tests/test_contextual_bandit.py](polyguard-rl/tests/test_contextual_bandit.py), [tests/test_policy_schema.py](polyguard-rl/tests/test_policy_schema.py), [tests/test_provider_runtime.py](polyguard-rl/tests/test_provider_runtime.py), [tests/test_postsave_inference.py](polyguard-rl/tests/test_postsave_inference.py), [tests/test_checkpoint_integrity.py](polyguard-rl/tests/test_checkpoint_integrity.py) |
+| API and product tooling | [tests/test_api.py](polyguard-rl/tests/test_api.py), [tests/test_medication_alternatives.py](polyguard-rl/tests/test_medication_alternatives.py), [tests/test_remote_env.py](polyguard-rl/tests/test_remote_env.py) |
+| Data and evidence | [tests/test_parser.py](polyguard-rl/tests/test_parser.py), [tests/test_dataops_parser.py](polyguard-rl/tests/test_dataops_parser.py), [tests/test_graph_infer.py](polyguard-rl/tests/test_graph_infer.py), [tests/test_submission_evidence.py](polyguard-rl/tests/test_submission_evidence.py) |
+| Submission, notebook, and HF flow | [tests/test_acceptance_gate.py](polyguard-rl/tests/test_acceptance_gate.py), [tests/test_runner_notebook.py](polyguard-rl/tests/test_runner_notebook.py), [tests/test_hf_training_sweep.py](polyguard-rl/tests/test_hf_training_sweep.py) |
+
+Additional architecture diagrams:
+
+- [System architecture](polyguard-rl/docs/assets/diagrams/system_architecture.png)
+- [Runtime step flow](polyguard-rl/docs/assets/diagrams/runtime_step_flow.png)
+- [Data and training pipeline](polyguard-rl/docs/assets/diagrams/data_training_pipeline.png)
+- [Multi-agent orchestration](polyguard-rl/docs/assets/diagrams/multi_agent_orchestration.png)
+- [Reward decomposition](polyguard-rl/docs/assets/diagrams/reward_decomposition.png)
+- [Episode state machine](polyguard-rl/docs/assets/diagrams/episode_state_machine.png)
+- [Evidence generation flow](polyguard-rl/docs/assets/diagrams/evidence_generation_flow.png)
+- [Deployment topology](polyguard-rl/docs/assets/diagrams/deployment_topology.png)
+- [Frontend runtime surface](polyguard-rl/docs/assets/diagrams/frontend_runtime_surface.png)
+
+## Limitations
+
+PolyGuard is a simulator and research environment. Its current data substrate
+is compact and intentionally inspectable, not a production clinical knowledge
+base. The final evidence set is strongest for Qwen 3B because that run has
+complete SFT, GRPO, post-save GRPO, policy-ablation, adapter, and checkpoint
+evidence. Qwen 0.5B and 1.5B have SFT reports/histories and post-save SFT
+evidence, but their adapter directories are marked `reports_only_or_partial` in
+the final manifest.
+
+The reward model is hand-designed and auditable. That is a feature for this
+OpenEnv setting, but it also means reward-channel design should be
+stress-tested as the data grows. The current ablations show that contextual
+bandits are useful and inspectable, while the `llm+bandit` combined stack needs
+more tuning to avoid loop-like failure behavior in some settings.
+
+The right conclusion is not "this is a clinical decision system." The right
+conclusion is that constrained environment feedback, verifier-backed rewards,
+agentic evaluation, and explicit failure mining are a better substrate for
+safety-critical medication-policy learning than free-form prompt responses.
+
+## References
+
+- Alexandre Larouche, Audrey Durand, Richard Khoury, Caroline Sirois.
+  [Neural Bandits for Data Mining: Searching for Dangerous Polypharmacy](https://arxiv.org/abs/2212.05190).
+  arXiv:2212.05190.
+- World Health Organization.
+  [Medication Without Harm](https://www.who.int/initiatives/medication-without-harm).
+- CDC.
+  [FastStats: Medication Safety Data](https://www.cdc.gov/medication-safety/data-research/facts-stats/index.html).
+- Shehab N, Lovegrove MC, Geller AI, Rose KO, Weidle NJ, Budnitz DS.
+  [US Emergency Department Visits for Outpatient Adverse Drug Events, 2013-2014](https://jamanetwork.com/journals/jama/fullarticle/2585977).
+  JAMA. 2016;316(20):2115-2125.
+- AHRQ / NCBI Bookshelf.
+  [Deprescribing To Reduce Medication Harms in Older Adults](https://www.ncbi.nlm.nih.gov/books/NBK600387/).
+- American Geriatrics Society.
+  [2023 updated AGS Beers Criteria for potentially inappropriate medication use in older adults](https://pmc.ncbi.nlm.nih.gov/articles/PMC12478568/).
+- O'Mahony et al.
+  [STOPP/START criteria for potentially inappropriate prescribing in older people: version 3](https://pmc.ncbi.nlm.nih.gov/articles/PMC10447584/).
+
+## License
+
+The project package declares an MIT license in
+[polyguard-rl/pyproject.toml](polyguard-rl/pyproject.toml). See
+[polyguard-rl/LICENSE](polyguard-rl/LICENSE) for the license text.