chore: sync local CyberSecurity_OWASP with HF Space history

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

docs/deep-research-report.md

@@ -0,0 +1,531 @@
+# OpenEnv Hackathon Execution Pipeline for a Safe Cybersecurity Analyst Environment
+
+## Executive decision and scope selection
+
+**SECTION 1 — Executive Decision**
+
+[SOURCED] The hackathon’s *validator + judging* constraints strongly favour environments that: (a) simulate a real-world task (not “games/toys”), (b) are fully OpenEnv-compliant, (c) ship with **≥3 tasks with graders**, (d) produce **scores/rewards in the 0–1 range**, (e) include a reproducible `inference.py` that uses the **OpenAI client** (for any LLM calls) and prints strict `[START]/[STEP]/[END]` logs, and (f) run within a ~**20 minute** budget on ~**2 vCPU / 8 GB** infra. citeturn3view6turn3view7turn22view1turn22view2  
+
+[INFERENCE] Under these realities, your cybersecurity-analyst direction is *not* automatically the best, but it *can* become a high-probability-to-win choice if—and only if—you narrow to a deterministic, bounded, “investigate → cite evidence → verify → remediate” loop where (i) tools are tightly sandboxed, (ii) graders are deterministic, and (iii) the action space is small enough to be learnable and demo-able under the runtime cap.
+
+[PROPOSAL] **Decision:** keep the cybersecurity direction, but **narrow aggressively** to a V1 environment that benchmarks **disciplined security triage + evidence-grounded reporting**, not pentesting/exploitation. The V1 I recommend building is:
+
+[PROPOSAL] **“SecOps Evidence Gym”** — a safe, isolated OpenEnv environment where an agent investigates a *synthetic* microservice “organisation” via a **bounded tool API**, collects **evidence IDs**, validates candidate findings through **deterministic verifiers**, and submits a structured remediation report.  
+
+[SOURCED] This matches strong “winner DNA” seen in `kube-sre-gym` (realistic professional workflow + verification + narrative clarity) while remaining implementable in a hackathon budget. citeturn10view0turn18view0  
+
+[PROPOSAL] **What to cut entirely in V1 (non-negotiable):**  
+- “Live target” behaviour; no external network targets; no arbitrary HTTP to the internet. 🔒  
+- Any exploit payload recipes, exploit chains, privilege-escalation playbooks, or “how to hack X”. 🔒  
+- Arbitrary shell access (`bash`, `kubectl`, `nmap`, etc.) inside the environment. (Action space explosion + safety risk.)  
+- LLM-only grading/judging for correctness. (Reward hacking + non-determinism.)  
+
+[SOURCED] **What to keep (but narrow):** tool-using investigation, multi-step interaction, and deterministic verification—these are consistent with what OpenEnv is designed to support (typed `reset/step/state`, isolated server, type-safe schemas). citeturn18view0turn19search1  
+
+**SECTION 3 — Candidate Scope Comparison**
+
+[SOURCED] The scoring below is anchored on hackathon validator requirements (3+ graded tasks, 0–1 scoring, strict inference logging, runtime limits) plus OpenEnv’s scaffolding/CLI/deployment model. citeturn3view6turn18view0turn22view1  
+
+[PROPOSAL] Weighted criteria (sum=1.00): judging fit 0.14, OpenEnv fit 0.12, grader determinism 0.14, implementation risk 0.12, runtime feasibility 0.10, demoability 0.10, real-world usefulness 0.10, novelty 0.08, training usefulness 0.06, shipping-on-time likelihood 0.04.
+
+| Candidate scope | Judging fit | OpenEnv fit | Determinism | Impl risk (lower=better) | Runtime | Demo | Real-world use | Novelty | Training use | Ship-on-time | Weighted total |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| **A. Your original direction:** “disciplined cyber analyst investigating a sandbox” (broad) | 8 | 7 | 6 | 4 | 6 | 8 | 8 | 8 | 8 | 4 | **6.7** |
+| **B. Narrow cyber variant (recommended):** evidence-first triage lab with bounded tools + deterministic verifiers + structured report | 9 | 8 | 9 | 7 | 9 | 9 | 8 | 7 | 8 | 8 | **8.4** |
+| **C. Adjacent: SRE incident triage (single-turn, deterministic logs → RCA)** | 9 | 8 | 10 | 9 | 10 | 8 | 8 | 5 | 6 | 9 | **8.3** |
+| **D. Adjacent: prompt-injection “WAF” benchmark** | 7 | 8 | 8 | 7 | 9 | 9 | 7 | 7 | 6 | 7 | **7.6** |
+
+[INFERENCE] Candidate C (SRE triage) is extremely validator-safe (many examples already pass deep validation), but it is likely more saturated and less “new” in judging. Candidate B keeps your cybersecurity theme while retaining the determinism and boundedness that the validator and time budget demand, so it is the best balance for **winning + real-world usefulness**.
+
+**SECTION 4 — Final V1 Problem Statement**
+
+[PROPOSAL] **One-sentence version:** Build a safe OpenEnv environment that trains and benchmarks an agent to perform **evidence-grounded security triage** on a bounded synthetic system and produce a **remediation-oriented report** without hallucinating.
+
+[PROPOSAL] **Short pitch (demo-ready):** “SecOps Evidence Gym” gives the model an alert and a constrained set of investigation tools. The model must gather evidence, validate findings with deterministic verifiers, and submit a structured report. Scores reward verified correctness, penalise hallucinated claims and wasted steps, and remain strictly within (0,1) to satisfy the hackathon validator. ✅🔒
+
+[PROPOSAL] **Precise implementation version:** A FastAPI OpenEnv server exposing typed `reset()`, `step()`, `state()` and a manifest `openenv.yaml` with at least three tasks (easy/medium/hard), each associated with a grader. The environment implements multi-step tool calls inside `step()`, and uses deterministic verifiers plus a strict score clamp to `(0,1)` for validator compatibility. citeturn18view0turn19search1turn23view0turn26view0turn27search0  
+
+## Winner patterns and judging fit extraction
+
+**SECTION 2 — Winner Pattern Extraction**
+
+[SOURCED] `kube-sre-gym` (OpenEnv Hackathon SF winner) demonstrates several patterns that appear strongly aligned with what judges value: a **realistic professional task**, an explicit **multi-step workflow** (triage → investigate → fix → verify), **multi-layer verification** (programmatic health checks + judge), a strong narrative that explains learning dynamics, and deployment on **Hugging Face Spaces**. citeturn10view0turn14view0  
+
+image_group{"layout":"carousel","aspect_ratio":"16:9","query":["kube-sre-gym OpenEnv Hackathon winner screenshot","OpenEnv framework architecture diagram","Hugging Face Spaces OpenEnv environment web demo","Incident Triage Environment OpenEnv Hackathon 2026 screenshot"],"num_per_query":1}
+
+[SOURCED] Concretely, `kube-sre-gym` highlights: “real cluster/tool interaction”, verification layers to prevent false success, curriculum progression, and strong documentation that makes the environment’s value obvious quickly. citeturn10view0  
+
+[SOURCED] A 2026 hackathon submission that explicitly claims Phase-2 deep-validation success (`Incident-Triage-Environment`) reveals particularly transferable “validator-winning” patterns:  
+- A manifest with `spec_version`, `runtime`, `app`, `port`, and a **`tasks:` list where each task has a `grader:` pointing to importable Python functions**. citeturn23view0  
+- Deterministic graders that clamp outputs to a validator-friendly range. citeturn26view0turn26view3  
+- An `inference.py` that uses the OpenAI SDK and prints the strict stdout protocol with `[START]`, `[STEP]`, `[END]` lines in a stable key=value format. citeturn22view1turn22view2turn22view4  
+
+[SOURCED] The official OpenEnv repo reinforces what is transferable: type-safe action/observation/state contracts, containerised isolation, and standard Gym-like APIs. It also explicitly says OpenEnv is experimental and APIs can change, which increases the value of a minimal, validator-first build loop. citeturn18view0turn19search2  
+
+[INFERENCE] Given hackathon evaluation combines programmatic checks with LLM scoring, you must optimise for **deterministic correctness** *and* a compelling narrative/demo. citeturn3view7turn3view6  
+
+[PROPOSAL] Transferable “winner patterns” you should copy (selectively):  
+- **Strong “professional workflow” framing** (SRE, security analyst, triage) with clear step boundaries.  
+- **Small, discoverable tool set** that mirrors real practice (logs, config, policy checks) while staying bounded.  
+- **Deterministic verification** (programmatic checks) as the source of truth for correctness.  
+- **Narrative traceability**: logs, episode IDs, and a short “watch the agent work” demo.  
+- **Deployment excellence**: clean Docker build, working `/health`, working `/web` UI if enabled, and reproducible inference.
+
+[PROPOSAL] What *not* to copy blindly:  
+- The “real cluster” dependency (e.g., GKE) is high operational burden and can fail the hackathon’s limited infra budget. citeturn10view0turn3view6  
+- LLM-as-judge for correctness (too easy to reward-hack; non-deterministic). (Use it, at most, for *format/style*, not correctness.)  
+
+## Core V1 environment design and benchmark tasks
+
+**SECTION 8 — Core Environment Design**
+
+**V1 concept (aggressively narrow).**  
+[PROPOSAL] Your environment is a **synthetic organisation** with a small, fixed topology (three “services” + artefacts). The agent receives an alert. It can only interact via **approved tools** (implemented inside the simulator). It must (a) gather evidence IDs, (b) validate candidate findings, and (c) submit a report.
+
+**Topology (V1).**  
+[PROPOSAL] Fixed components (no containers inside containers):  
+- `gateway` (public entry), `profile-service`, `admin-service`  
+- `repo_snapshot` (static code/config excerpts)  
+- `telemetry` (sanitised logs + “header snapshot” + “dependency manifest snapshot”)  
+
+**Reset logic.**  
+[PROPOSAL] `reset(task_id=..., seed=...)` selects a scenario variant and initialises:  
+- episode ID, step count  
+- scenario ground truth (one injected issue per episode in V1)  
+- tool budgets + “allowed scope” banner  
+- an evidence registry mapping `EVID-### → artefact snippet`  
+Return an initial observation containing the alert, the tool catalogue, and an empty “verified findings” list.
+
+**Randomisation strategy.**  
+[PROPOSAL] Use seed-driven, deterministic randomisation:  
+- rename services/routes/IDs (`profile-service` might become `user-profile`),  
+- shuffle benign log lines around the key evidence,  
+- vary exact header sets / dependency versions within a small closed set,  
+- keep each scenario **fully reproducible from the seed**.
+
+[SOURCED] Benchmark generators (e.g., AMaze) exist specifically to create diverse but controlled environments for evaluating generalisation, supporting the idea of seeded procedural variation rather than a single static scenario. citeturn16search7turn16search1  
+
+**Safety boundaries.**  
+[PROPOSAL] The sandbox contains **no live targets**, no real secrets, and no outbound network. “Secrets” are synthetic strings with an explicit “DO NOT USE OUTSIDE LAB” marker. Tools return synthetic results only. 🔒  
+
+[SOURCED] NIST’s cyber range guidance emphasises cyber ranges as safe and legal environments for training and assessment; separate research also discusses that cyber ranges themselves have security risks that must be mitigated (e.g., leakage/misuse), reinforcing the need for strict isolation and artefact sanitisation. citeturn29search1turn29search2  
+
+**How state is exposed to the agent.**  
+[PROPOSAL] Expose only a concise state summary: current phase, step budget remaining, tools remaining, verified findings count, and recent evidence IDs. Keep full ground truth hidden.
+
+**Tool/action design (bounded action space).**  
+[PROPOSAL] V1 tool list (keep it ≤8 tools):  
+1) `list_assets()` → returns asset IDs and route IDs  
+2) `get_log_events(service_id, query)` → returns evidence IDs  
+3) `check_security_headers(service_id)` → returns evidence IDs + pass/fail list  
+4) `search_repo(query)` → returns evidence IDs from code snippets  
+5) `scan_dependencies()` → returns evidence IDs from a lockfile excerpt  
+6) `create_finding(finding_type, evidence_ids, severity_guess, remediation)` → stores candidate finding  
+7) `validate_finding(finding_id)` → deterministic verifier; returns `(verified, matching_gt_id)`  
+8) `submit_report(report_json)` → terminal action  
+
+**Anti-loop logic.**  
+[PROPOSAL] Track action signatures `(tool_name, args_hash)` and:  
+- apply increasing penalties for repeats,  
+- hard-stop an episode if identical actions repeat ≥6 times, returning `done=True` with a low score,  
+- always return a valid observation (never a server crash) to preserve training rollouts.
+
+[SOURCED] OpenEnv’s environment-creation guidance strongly implies you should implement robust behaviour around `reset/step/state` with typed contracts and predictable server behaviour. citeturn19search1turn18view0  
+
+**SECTION 9 — Tasks / Benchmarks**
+
+[SOURCED] The hackathon requires **at least 3 tasks with graders** and explicitly checks the tasks registry. citeturn3view6turn27search0  
+
+[PROPOSAL] V1 ships exactly **3 flagship tasks**, difficulty-tiered, each with deterministic success criteria and intermediate milestones.
+
+**Flagship tasks (easy/medium/hard).**  
+[PROPOSAL] Each task is a *family* with small seeded variants.
+
+**Easy: Secret exposure in repo snapshot**  
+- Goal: identify a leaked synthetic API key in a config file excerpt; propose rotation/removal.  
+- Deterministic success: report includes the correct finding type `secret_exposure`, includes ≥1 correct evidence ID, and remediation mentions rotation + removal.  
+- Intermediate rewards: `search_repo()` surfaces the evidence ID; `create_finding()` with correct type gets partial credit; `validate_finding()` confirms.  
+- False-positive check: claiming *additional* vulnerabilities not verified triggers penalty.
+
+**Medium: Missing security headers**  
+- Goal: detect missing/weak security headers in a service “header snapshot”; propose remediation.  
+- Deterministic success: correct missing header set identification (from a fixed list), plus remediation mapping (e.g., add HSTS, CSP) within the environment’s rubric.  
+- Intermediate rewards: correct tool usage (`check_security_headers()`), correct mapping to finding type, successful verifier validation.  
+- Generalisation: header ordering/extra benign headers vary by seed.
+
+**Hard: Authorisation boundary misconfiguration**  
+- Goal: detect an access control policy bug in a route/role matrix (modelled safely, without exploitation).  
+- Deterministic success: evidence IDs must show the policy mismatch; report must describe impact and remediation (principle of least privilege + policy fix + regression test).  
+- Intermediate rewards: `list_assets()` + `get_log_events()` reveal the mismatch pattern; candidate finding validated.  
+- False-positive guardrail: generic “SQLi/RCE” claims penalised unless evidence supports (it won’t, by design).
+
+**Stretch tasks (post-V1, not for hackathon critical path).**  
+[PROPOSAL] Dependency-risk identification (synthetic CVE mapping), error-handling info leak, prioritisation under strict budget, and multi-finding episodes (2 findings) — but only once the validator-safe V1 is shipped.
+
+## OpenEnv compliance blueprint and repo plan
+
+**SECTION 6 — OpenEnv Compliance Blueprint**
+
+[SOURCED] OpenEnv’s core contract is Gymnasium-like APIs (`reset()`, `step()`, `state()`), with type-safe models, packaged behind a FastAPI server and typically accessed via an EnvClient. citeturn18view0turn19search1  
+
+[SOURCED] For environment creators, OpenEnv explicitly supports `openenv init`, and documents a canonical structure: `models.py`, `client.py`, `server/app.py`, `server/<environment>.py`, plus `openenv.yaml` and packaging metadata. citeturn18view0turn18view1  
+
+[SOURCED] OpenEnv provides CLI commands including `openenv init` and `openenv push` for deploying to **Hugging Face Spaces**. citeturn18view0turn17view0  
+
+[SOURCED] The OpenEnv repo’s environment-building guide demonstrates typed models (Action/Observation/State) as Python dataclasses and a `create_fastapi_app(...)` helper to serve the environment. citeturn19search1  
+
+[SOURCED] The OpenEnv repo explicitly warns *not* to copy outdated manifest patterns; current examples use `spec_version`, `type`, `runtime`, `app`, `port`. citeturn19search2turn23view0  
+
+**Validator-sensitive details you must implement (non-negotiable).**  
+[PROPOSAL] Based on official requirements + observed validator behaviour:  
+- Provide `openenv.yaml` with `spec_version: 1`, `name`, `runtime: fastapi`, `app: server.app:app`, `port: <int>`, and a `tasks:` list with **≥3 tasks each having `id`, `description`, `grader`**. citeturn23view0turn19search2  
+- Ensure each task’s final score is **strictly within (0,1)** to avoid fail-fast validation errors. citeturn27search0turn26view0  
+- Implement an `inference.py` that prints `[START]/[STEP]/[END]` lines exactly and uses the OpenAI SDK for LLM calls (if any), reading `HF_TOKEN`, `API_BASE_URL`, `MODEL_NAME`. citeturn3view6turn22view1turn22view2  
+- Provide a `/health` endpoint that returns 200 once ready (commonly used in examples and deployment docs). citeturn17view0turn20view0  
+
+**Sync vs async.**  
+[SOURCED] OpenEnv supports async-first clients with a `.sync()` wrapper for synchronous usage. For hackathon inference scripts, synchronous control flow is often simpler and widely used in examples. citeturn18view0turn22view4  
+
+**What not to copy from older examples.**  
+[SOURCED] Some course material shows a simplified `openenv.yaml` (`name/version/description`), but the repo’s skill guidance explicitly warns against outdated manifests; follow the current spec-style manifest used in validated examples. citeturn19search2turn19search11turn23view0  
+
+**SECTION 7 — Repo / File Tree Plan**
+
+[SOURCED] OpenEnv’s scaffold and common community submissions converge on a predictable repository layout and file naming. citeturn18view0turn20view0turn23view0  
+
+[PROPOSAL] Recommended repo structure (submission-ready):
+
+```
+secops_evidence_gym/
+  openenv.yaml                 # REQUIRED: spec_version, runtime, app, port, tasks+graders
+  pyproject.toml               # REQUIRED: package metadata + deps
+  README.md                    # REQUIRED: judging narrative + quickstart + safety boundaries
+  inference.py                 # REQUIRED: strict stdout logs + OpenAI client usage
+  models.py                    # REQUIRED: typed Action/Observation/State dataclasses
+  client.py                    # REQUIRED: EnvClient wrapper (sync + async)
+  __init__.py                  # REQUIRED: export Env + models for pip install
+
+  server/
+    app.py                     # REQUIRED: create_fastapi_app(...) wiring + /health
+    environment.py             # REQUIRED: SecOpsEvidenceGymEnvironment(reset/step/state)
+    graders.py                 # REQUIRED: grade_easy/medium/hard + safe_reward clamp
+    tasks.py                   # OPTIONAL (high-leverage): scenario registry + seed sampling
+    safety.py                  # OPTIONAL (high-leverage): tool allowlist + sanitisation helpers
+    requirements.txt           # OPTIONAL (if Docker build uses it)
+    Dockerfile                 # REQUIRED (practically): HF Spaces docker build
+
+  tests/
+    test_api_contract.py       # smoke: reset/step/state doesn’t crash; reward range
+    test_graders.py            # unit: deterministic scoring + strict (0,1) clamp
+    test_seed_determinism.py   # unit: same seed → same evidence IDs
+```
+
+[PROPOSAL] Mandatory for hackathon success: `openenv.yaml`, server app wiring, three tasks+graders, Docker build success, `inference.py` with strict logs, and a README that makes the environment’s value obvious in <60 seconds.
+
+## Reward, grading, and anti-hallucination design
+
+**SECTION 10 — Reward Design**
+
+[SOURCED] OpenEnv leaves reward semantics to the environment; you are responsible for correctness scoring and determinism. citeturn18view0turn19search1  
+
+[SOURCED] Hackathon validation has shown strict “score must be between 0 and 1 (not 0.0 and not 1.0)” behaviour, and teams clamp rewards (e.g., 0.01–0.99). citeturn27search0turn26view0  
+
+[SOURCED] Empirical RL research in other domains (e.g., autonomous racing) shows reward design choices materially affect performance and generalisation, supporting the need for careful shaping rather than a single sparse terminal reward. citeturn15view2  
+
+[PROPOSAL] **Core principle:** correctness is **verifier-gated**, not language-judged. You can optionally add *format/style* checks, but never allow style to dominate correctness reward.
+
+### Reward structure (practical V1)
+
+[PROPOSAL] Normalise the final *task score* into `(0,1)` and keep per-step rewards small enough that summed episode reward stays in `(0,1)` as well (or only final reward is used, depending on your environment semantics). Use a single “score” to satisfy the validator and expose detailed breakdowns in `observation.metadata`.
+
+**Terminal (sparse) components** ✅  
+[PROPOSAL]  
+- `+0.60` if at least one ground-truth finding is verified and correctly described (type + impact).  
+- `+0.15` if the report includes **≥1 valid evidence ID** per finding and those IDs correspond to the right artefacts.  
+- `+0.15` if remediation is actionable (specific control, config, test).  
+- `-0.40` per hallucinated/unverified finding claimed in the report.  
+- `-0.20` if the agent fails to run `validate_finding()` before `submit_report()`.
+
+**Intermediate (dense) components** 🧭  
+[PROPOSAL]  
+- `+0.02` for discovering a *new* relevant evidence ID (first time only).  
+- `+0.03` for creating a well-formed candidate finding that references evidence IDs.  
+- `-0.01` per step (efficiency pressure).  
+- `-0.03` for repeating the same tool call (exact same args) beyond 2 times.  
+
+**False-positive penalties / anti-hallucination** 🧯  
+[PROPOSAL] A “hallucination” is operationally defined as: the report asserts a finding that is not in the environment’s `verified_findings` list. This is easy to compute deterministically and maps directly to your stated goal (“avoid hallucinating findings”).
+
+### Avoiding reward hacking
+
+[PROPOSAL] Hardening rules:  
+- Cap rewards from verbosity: extra words do not add points.  
+- Make evidence IDs required for high scores (prevents purely rhetorical “security speak”).  
+- Penalise calling `validate_finding()` repeatedly without new evidence.  
+- Reject “kitchen sink” reporting by penalising extra unverified findings.
+
+### Binary vs shaped reward
+
+[PROPOSAL] **Binary-only** (0/1) will be easy to implement but brittle for multi-step tool use; the agent gets no gradient for *how* to investigate efficiently.  
+
+[PROPOSAL] **Lightly shaped** (recommended) keeps correctness deterministic while providing enough signal to train investigation workflow (evidence collection, validation order, loop avoidance). This mirrors the broader lesson from reward engineering research: shaping and tuning can significantly alter learning outcomes. citeturn15view2  
+
+### Deterministic judge vs hybrid judge
+
+[PROPOSAL]  
+- **Strict deterministic judge (recommended V1):** all correctness via verifiers + string/structure checks.  
+- **Hybrid (stretch):** add a small LLM-based style score (e.g., clarity), heavily downweighted (≤0.05 of total) and never affecting pass/fail correctness.
+
+## Baseline inference pipeline and strict stdout logging
+
+**SECTION 11 — Baseline Inference Pipeline**
+
+[SOURCED] Hackathon requirements include: a reproducible `inference.py`, the OpenAI client requirement for LLM calls (using provided env vars), and strict stdout logging. citeturn3view6  
+
+[SOURCED] A concrete, hackathon-aligned stdout format has been used by validated submissions (example):  
+- `[START] task=<name> env=<benchmark> model=<model_name>`  
+- `[STEP] step=<n> action=<str> reward=<0.00> done=<true|false> error=<msg|null>`  
+- `[END] task=<name> success=<true|false> steps=<n> score=<0.00> rewards=<r1,r2,...>` citeturn22view1turn22view2  
+
+[SOURCED] The same example inference uses the OpenAI SDK, reading `API_BASE_URL`, `MODEL_NAME`, and `HF_TOKEN`. citeturn22view1turn22view4  
+
+### Responsibilities of `inference.py`
+
+[PROPOSAL] `inference.py` should:  
+- read env vars: `HF_TOKEN`, `API_BASE_URL`, `MODEL_NAME`, `ENV_URL` (and optionally `TASK_NAME` override),  
+- connect to the env via `.sync()` client,  
+- run tasks in a fixed order (easy → medium → hard),  
+- execute a bounded number of steps per task,  
+- log exactly one `[START]...` per task, one `[END]...` per task, and a `[STEP]...` per environment step,  
+- always exit with code 0 (even on failures) and log errors in the `[STEP] error=` field to avoid hard crashes.
+
+### Control flow (V1 baseline strategy)
+
+[PROPOSAL] Use a **hybrid baseline** that is reliable under time constraints:  
+- scripted tool sequence per task (fast, deterministic),  
+- one LLM call (optional) to draft the final report from gathered evidence (so the demo shows “agentic reasoning”),  
+- temperature fixed to 0 for reproducibility (and lower variance).  
+
+[SOURCED] Deterministic inference settings like `TEMPERATURE=0.0` are used in competitive OpenEnv hackathon baselines. citeturn20view0turn22view4  
+
+### Minimum viable baseline (must ship)
+
+[PROPOSAL] For each task:  
+1) `reset(task_id=<tier>)`  
+2) run 2–4 tool calls that are always relevant (e.g., `check_security_headers`, `search_repo`, etc.)  
+3) `create_finding(...)` using evidence IDs  
+4) `validate_finding(finding_id)`  
+5) `submit_report(report_json)`  
+
+### Stronger baseline (only if time permits)
+
+[PROPOSAL] Add one planning LLM call that chooses among tools based on the alert type, but still keep a hard step limit, and always include verifier validation before reporting.
+
+## Complete build, validation, deployment, and submission pipeline
+
+**SECTION 5 — Complete End-to-End Pipeline**
+
+[SOURCED] This pipeline is built to satisfy both OpenEnv conventions (init/push, typed models, FastAPI server) and hackathon validation constraints (tasks/graders, inference logging, runtime budgets). citeturn18view0turn19search2turn3view6turn22view1  
+
+### Phase goals, deliverables, verification (execution-ready)
+
+[PROPOSAL] The table below is the “do-this-in-order” execution plan. It is intentionally validator-first.
+
+| Phase | Goal | Deliverables | Files touched | Acceptance criteria | Main risks | How to verify |
+|---|---|---|---|---|---|---|
+| Scope lock | Freeze V1 to 3 tasks + bounded tools | 1-page spec + non-goals | README.md | No pentest/exploit scope; 3 tasks defined | Scope creep | Manual checklist |
+| Scaffold | Generate OpenEnv skeleton | Working importable package | openenv.yaml, models.py, client.py, server/* | `python -c "import ..."` succeeds | Wrong template/paths | Local import smoke test |
+| Environment core | Implement reset/step/state; tool router | Simulator runs end-to-end | server/environment.py | reset+step returns typed observation; no crashes | Action validation crashes | manual `curl` + python client |
+| Tasks + graders | Implement 3 graders + strict (0,1) clamp | `grade_easy/medium/hard` | server/graders.py, openenv.yaml | tasks discoverable; scores strictly in (0,1) | Validator fail-fast | unit tests + manual checks |
+| Baseline inference | Make inference reproducible + strict logs | inference.py | inference.py | prints correct `[START]/[STEP]/[END]` | log-parser failure | run script locally |
+| Local validation | Run OpenEnv build & validate | passes `openenv validate` | Dockerfile, server/app.py | validate passes locally | port mismatch | `openenv validate --url ...` |
+| Docker + HF | Deploy to Spaces | live endpoint | openenv push output | `/health` 200; reset+step works remotely | HF port/env mismatch | curl + python client |
+| Submission | Final narrative + demo | polished README + screenshots | README.md | demo works in <2 min | unclear story | run “demo script” |
+
+### Concrete build plan with commands
+
+[SOURCED] OpenEnv supports `openenv init` and `openenv push` and documents this as the standard creator workflow. citeturn18view0turn17view0  
+[SOURCED] The OpenEnv course also provides a grounded dev loop: `uv sync`, `uv run server`, `curl /health`, and Docker build/run commands. citeturn17view0  
+
+[PROPOSAL] Commands (copy/paste order):
+
+1) **Scaffold**
+```bash
+pip install openenv-core
+openenv init secops_evidence_gym
+cd secops_evidence_gym
+```
+[SOURCED] `openenv init` is the documented way to scaffold a new environment. citeturn18view0turn18view2  
+
+2) **Local dev install + run**
+```bash
+uv sync
+uv run server
+curl http://localhost:8000/health
+```
+[SOURCED] `uv run server` and `/health` checks are part of the recommended iteration loop in OpenEnv course materials. citeturn17view0  
+
+3) **Implement core files (edit)**
+- `models.py`: define `Action/Observation/State` dataclasses  
+- `server/environment.py`: implement reset/step/state + tool routing  
+- `server/graders.py`: implement `grade_easy/grade_medium/grade_hard` + `safe_reward()`  
+- `openenv.yaml`: add `tasks:` with grader import paths  
+
+[SOURCED] OpenEnv’s environment-building guide explicitly directs you to define models and implement `reset/step/state`, then wire a FastAPI app. citeturn19search1  
+[SOURCED] A validator-aligned `openenv.yaml` with `spec_version`, `runtime`, `app`, `port`, and `tasks` exists in deep-validation passing examples. citeturn23view0  
+
+4) **Build + validate (local)**
+```bash
+openenv build
+openenv validate --verbose
+```
+[SOURCED] `openenv build` and `openenv validate` are part of OpenEnv’s recommended validation workflow. citeturn19search2  
+
+5) **Docker build/run smoke test**
+```bash
+docker build -t secops-evidence-gym:latest -f server/Dockerfile .
+docker run -p 8000:8000 secops-evidence-gym:latest
+curl http://localhost:8000/health
+```
+[SOURCED] This `docker build -f server/Dockerfile .` pattern is directly shown in OpenEnv deployment course material. citeturn17view0  
+
+6) **Run inference locally**
+```bash
+export HF_TOKEN="..."
+export API_BASE_URL="..."
+export MODEL_NAME="..."
+export ENV_URL="http://localhost:8000"
+python inference.py
+```
+[SOURCED] These env var names and OpenAI SDK usage are consistent with hackathon guidance and existing inference implementations. citeturn3view6turn22view4  
+
+7) **Deploy to Hugging Face Spaces**
+```bash
+openenv push --repo-id <your-hf-username>/secops-evidence-gym
+```
+[SOURCED] `openenv push` is described as the fastest path to deploy to **Hugging Face Spaces**. citeturn17view0turn18view0  
+
+### Testing and validation plan (high-signal)
+
+[SOURCED] OpenEnv stresses predictable API behaviour and type-safe contracts; hackathon validation is fail-fast. citeturn18view0turn27search0  
+
+[PROPOSAL] Test layers (in priority order):  
+- **API contract smoke tests:** reset/step/state return valid JSON; never crash on invalid tool name (should return an observation with an error field).  
+- **Grader tests:** for each task, verify (a) correctness cases score high, (b) hallucination cases score low, (c) score always ∈ (0,1).  
+- **Seed determinism tests:** same `seed` produces same evidence IDs and same verifier outputs.  
+- **Runtime test:** run `inference.py` end-to-end and assert wall-clock < 2 minutes locally; assume < 20 minutes on grader infra even with cold starts. citeturn3view6turn22view4  
+- **Reward sanity tests:** ensure reward increases monotonically with verified correctness; fails if verbosity alone increases reward.
+
+## Submission packaging, execution roadmap, real-world usefulness, and failure modes
+
+**SECTION 14 — README / Demo / Submission Narrative**  
+[SOURCED] Judges likely assess both the environment’s technical correctness (programmatic checks) and qualitative merit (LLM scoring / narrative). citeturn3view7  
+
+[PROPOSAL] README structure that “feels like a winner” 🏆:  
+- **Hero block:** one-paragraph pitch + why it’s real-world + safety claim.  
+- **Two-minute demo:** copy/paste commands + expected output snippet with `[START]/[STEP]/[END]`.  
+- **Environment contract:** action schema, observation schema, task list.  
+- **Grading:** explain deterministic verifiers + hallucination penalties.  
+- **Safety & isolation:** explicit exclusions (no egress, no shell, synthetic artefacts).  
+- **Real-world relevance:** how this benchmarks/reporting maps to security workflows (triage, evidence, remediation).  
+- **Screenshots:** web UI (optional) + an evidence trace + one scored report example.  
+
+**SECTION 15 — Project Management Plan**  
+[PROPOSAL] Day-by-day (assuming a hackathon-style sprint):
+
+- **Day 0 (scope lock + scaffold):** environment skeleton, `openenv.yaml` with 3 tasks, stub graders returning 0.5 (clamped), server runs locally.  
+- **Day 1 (determinism + validator):** implement scenario generator, evidence registry, verifiers, and strict (0,1) scoring; pass `openenv validate`.  
+- **Day 2 (baseline + polish):** implement `inference.py` strict logs; deploy to Spaces; polish README + demo artefacts.
+
+[PROPOSAL] Critical path: `openenv.yaml tasks+graders` → grader clamp `(0,1)` → inference stdout format → Docker+Spaces deployment. (Everything else is secondary.)
+
+**SECTION 16 — Real-World Usefulness Plan**  
+[SOURCED] NIST’s testing guide emphasises planning, conducting tests, analysing findings, and developing mitigation strategies; your environment’s “evidence → remediation” focus aligns with that lifecycle without requiring offensive exploitation. citeturn29search8turn29search0  
+
+[PROPOSAL] Who would care after the hackathon:  
+- security engineering teams evaluating agentic “triage + reporting” reliability,  
+- LLM tooling teams wanting benchmarks for **non-hallucinating, evidence-grounded** outputs,  
+- training teams building safe cyber ranges (without weaponisation).
+
+[PROPOSAL] Post-hackathon upgrades (highest leverage):  
+- export trajectories as JSONL for offline training,  
+- add more scenario families (still safe) and a held-out split for generalisation,  
+- integrate with RL trainers (e.g., TRL’s OpenEnv integration) to show real training curves. citeturn19search6turn10view0  
+
+[SOURCED] PenGym provides evidence that realism/faithfulness of environments can affect transfer and stability when moving from simulation to more realistic settings—so you should roadmap a “higher fidelity mode” (still safe) later, not in V1. citeturn15view0  
+
+**SECTION 17 — Why the naive version would fail**  
+[PROPOSAL] Top failure patterns (and why they kill submissions):  
+- Too broad (full cyber range, live services): fails time/infra constraints. citeturn3view6turn10view0  
+- Fuzzy grading (LLM-only judging): non-deterministic, easy to game.  
+- Unbounded tools (shell/network): unsafe + untrainable action space.  
+- Scores at exactly 0.0 or 1.0: fail-fast “out of range” validator. citeturn27search0turn26view0  
+- Inference logs not parseable: phase-1 failure even if env is good. citeturn3view6turn22view1  
+- Port / health issues on Spaces: container “works locally” but fails remotely. citeturn17view0turn20view0  
+
+**SECTION 18 — Final Recommendation**
+
+[PROPOSAL] **What should you build?**  
+Build **SecOps Evidence Gym**: a deterministic, safe, sandbox-only cyber analyst environment focused on evidence collection, verifier validation, and remediation reporting.
+
+[PROPOSAL] **What should V1 include? (minimum winning set)**  
+- OpenEnv-compliant FastAPI env with typed models and `reset/step/state`. citeturn18view0turn19search1  
+- `openenv.yaml` with **3 tasks + graders**. citeturn23view0turn3view6  
+- Deterministic verifiers + strict score clamp to `(0,1)`. citeturn27search0turn26view0  
+- Baseline `inference.py` with strict `[START]/[STEP]/[END]` logging + OpenAI SDK usage for any LLM calls. citeturn3view6turn22view1turn22view4  
+- HF Spaces deployment with a working `/health`. citeturn17view0turn20view0  
+
+[PROPOSAL] **What should you cut?**  
+- Any real pentesting/offensive content, any arbitrary command execution, any live targets, any correctness scoring via an LLM judge.
+
+[PROPOSAL] **Top 5 implementation decisions that matter most**  
+1) Validator-safe `openenv.yaml` tasks+graders wiring. citeturn23view0  
+2) Score/range compliance: clamp to `(0,1)` everywhere. citeturn27search0turn26view0  
+3) Strict stdout format in `inference.py`. citeturn22view1turn22view2  
+4) Deterministic verifiers as the source of truth.  
+5) Bounded tool set (≤8 tools) with anti-loop penalties.
+
+[PROPOSAL] **Minimum viable winning submission**  
+A V1 with 3 tasks, deterministic graders, bounded tools, strict inference logging, and a polished README + demo trace.
+
+[PROPOSAL] **Minimum viable real-world useful submission**  
+The same V1, plus: seed determinism, trajectory export, and a clear “how to add new scenarios” contributor guide.
+
+[PROPOSAL] **If you only have time for 20% of ambition—do this exact 20%:**  
+- Implement **one** robust multi-step loop (tools → validate → report)  
+- Implement **exactly 3** tasks (easy/medium/hard)  
+- Make graders deterministic and validator-safe  
+- Make deployment + inference bulletproof  
+Everything else is stretch.
+
+**Confidence (my estimate): 8.4/10** ✅🔥
+
+## Sources and credibility ratings (with exact links)
+
+[SOURCED] Ratings are my judgement of authority + relevance for this hackathon context (0–10). URLs are provided verbatim in code form.
+
+### Tier 1 (official OpenEnv + hackathon dashboard)
+- Credibility **9.5/10** — `https://github.com/meta-pytorch/OpenEnv` citeturn18view0  
+- Credibility **9.0/10** — `https://github.com/meta-pytorch/OpenEnv/blob/main/envs/README.md` citeturn19search1  
+- Credibility **8.5/10** — `https://github.com/meta-pytorch/OpenEnv/blob/main/.claude/skills/generate-openenv-env/SKILL.md` citeturn19search2  
+- Credibility **9.0/10** — `https://www.scaler.com/school-of-technology/meta-pytorch-hackathon/dashboard` citeturn1view0turn3view6turn3view7  
+
+### Tier 2 (strong community exemplars)
+- Credibility **8.5/10** — `https://github.com/sid-rp/kube-sre-gym` citeturn10view0  
+- Credibility **8.0/10** — `https://huggingface.co/openenv-community` citeturn14view0  
+- Credibility **7.5/10** — `https://github.com/Harikishanth/Incident-Triage-Environment` citeturn20view0turn23view0turn22view1  
+
+### Tier 3 (peer-reviewed / primary references for design constraints)
+- Credibility **8.5/10** — PenGym (Computers & Security, open access): `https://www.sciencedirect.com/science/article/pii/S0167404824004450` citeturn15view0  
+- Credibility **8.0/10** — Reward design + generalisation (Scientific Reports, 2025): `https://www.nature.com/articles/s41598-025-27702-6` citeturn15view2  
+- Credibility **8.5/10** — AMaze (JOSS, 2025): `https://joss.theoj.org/papers/10.21105/joss.07208` citeturn16search7  
+- Credibility **9.5/10** — NIST SP 800-115: `https://csrc.nist.gov/pubs/sp/800/115/final` citeturn29search8  
+- Credibility **9.0/10** — NIST “Cyber Range: A Guide” (PDF landing): `https://www.nist.gov/document/cyber-range` citeturn29search1  
+- Credibility **7.5/10** — “Cybersecurity of Cyber Ranges: Threats and Mitigations” (IJISR, 2022 PDF): `https://infonomics-society.org/wp-content/uploads/Cybersecurity-of-Cyber-Ranges.pdf` citeturn29search2  
+
+### Tier 4 (useful validator “ground truth” signals from the field)
+- Credibility **6.5/10** — Validator failure mode discussion (score must be strictly between 0 and 1): `https://www.reddit.com/r/pytorch/comments/1shi767/meta_x_pytorch_x_sst_x_openenv_hackathon_phase_2/` citeturn27search0  
+- Credibility **7.0/10** — Strict logging format reference via a verified submission’s `inference.py`: `https://github.com/Harikishanth/Incident-Triage-Environment/blob/main/inference.py` citeturn22view1turn22view2  
+
+### Uploaded reference you provided
+- Credibility **7.0/10** (useful as a design draft; not independently authoritative) — `deep-research-report (2).md` fileciteturn2file0

inference.py

@@ -0,0 +1,290 @@
+#!/usr/bin/env python3
+"""Model-backed baseline inference for the Cyber Analyst OpenEnv environment."""
+
+from __future__ import annotations
+
+import json
+import os
+import sys
+import textwrap
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any
+
+from openai import OpenAI
+
+PACKAGE_PARENT = Path(__file__).resolve().parent.parent
+if str(PACKAGE_PARENT) not in sys.path:
+    sys.path.insert(0, str(PACKAGE_PARENT))
+
+from Cyber_analyst import CyberAnalystAction, CyberAnalystEnv, CyberAnalystObservation
+
+
+ENV_NAME = "Cyber_analyst"
+ENV_URL = os.getenv("ENV_URL", "http://localhost:8000")
+API_BASE_URL = os.getenv("API_BASE_URL", "https://router.huggingface.co/v1")
+MODEL_NAME = os.getenv("MODEL_NAME", "google/gemma-4-31B-it:fastest")
+TEMPERATURE = float(os.getenv("TEMPERATURE", "0.0"))
+MAX_TOKENS = int(os.getenv("MAX_TOKENS", "512"))
+MAX_STEPS = int(os.getenv("MAX_STEPS", "12"))
+SEED = int(os.getenv("SEED", "7"))
+TASK_IDS = [
+    "secret_exposure_easy",
+    "missing_security_headers_medium",
+    "authz_boundary_hard",
+]
+
+SYSTEM_PROMPT = textwrap.dedent(
+    """
+    You are running a bounded Cyber Analyst benchmark. You may only choose one
+    tool call from the provided tool catalog per turn. All evidence is synthetic;
+    do not request shell access, live network access, or external investigation.
+
+    Return exactly one compact JSON object and no surrounding text:
+    {"tool_name":"<tool name>","args":{...}}
+
+    To complete an episode, first discover relevant evidence, then create and
+    validate a finding, then submit a report_json with findings that include
+    finding_type, evidence_ids, impact, and remediation.
+    """
+).strip()
+
+
+@dataclass(frozen=True)
+class LLMConfig:
+    base_url: str
+    model_name: str
+    temperature: float
+    max_tokens: int
+
+
+class ModelActionError(RuntimeError):
+    """Raised when the model cannot provide a valid benchmark action."""
+
+
+def build_llm_config() -> LLMConfig:
+    return LLMConfig(
+        base_url=API_BASE_URL,
+        model_name=MODEL_NAME,
+        temperature=TEMPERATURE,
+        max_tokens=MAX_TOKENS,
+    )
+
+
+def build_openai_client() -> OpenAI:
+    """Return an OpenAI-compatible client for the Hugging Face Router."""
+
+    return OpenAI(base_url=API_BASE_URL, api_key=os.environ["HF_TOKEN"])
+
+
+def single_line(value: str) -> str:
+    return " ".join(str(value).split())
+
+
+def action_to_log(action: CyberAnalystAction) -> str:
+    payload = {"tool_name": action.tool_name, "args": action.args}
+    return single_line(json.dumps(payload, sort_keys=True, separators=(",", ":")))
+
+
+def log_start(task_id: str, llm_config: LLMConfig) -> None:
+    print(
+        f"[START] task={task_id} env={ENV_NAME} model={llm_config.model_name}",
+        flush=True,
+    )
+
+
+def log_step(
+    step: int, action: CyberAnalystAction, reward: float | None, done: bool, error: str
+) -> None:
+    reward_value = 0.0 if reward is None else float(reward)
+    error_value = single_line(error) if error else "null"
+    print(
+        f"[STEP] step={step} action={action_to_log(action)} "
+        f"reward={reward_value:.2f} done={str(done).lower()} error={error_value}",
+        flush=True,
+    )
+
+
+def log_end(task_id: str, success: bool, steps: int, score: float, rewards: list[float]) -> None:
+    rewards_text = ",".join(f"{reward:.2f}" for reward in rewards)
+    print(
+        f"[END] task={task_id} success={str(success).lower()} "
+        f"steps={steps} score={score:.2f} rewards={rewards_text}",
+        flush=True,
+    )
+
+
+def observation_payload(obs: CyberAnalystObservation) -> dict[str, Any]:
+    return {
+        "task_id": obs.task_id,
+        "alert": obs.alert,
+        "phase": obs.phase,
+        "tool_catalog": obs.tool_catalog,
+        "tool_result": obs.tool_result,
+        "evidence_ids": obs.evidence_ids,
+        "candidate_findings": obs.candidate_findings,
+        "verified_findings": obs.verified_findings,
+        "step_budget_remaining": obs.step_budget_remaining,
+        "score_breakdown": obs.score_breakdown,
+        "error": obs.error,
+    }
+
+
+def build_user_prompt(task_id: str, step: int, obs: CyberAnalystObservation) -> str:
+    payload = {
+        "task_id": task_id,
+        "step": step,
+        "observation": observation_payload(obs),
+    }
+    return textwrap.dedent(
+        f"""
+        Choose the next benchmark tool call.
+        Current state JSON:
+        {json.dumps(payload, sort_keys=True)}
+        """
+    ).strip()
+
+
+def extract_json_object(text: str) -> dict[str, Any]:
+    content = text.strip()
+    if content.startswith("```"):
+        lines = content.splitlines()
+        if lines and lines[0].startswith("```"):
+            lines = lines[1:]
+        if lines and lines[-1].startswith("```"):
+            lines = lines[:-1]
+        content = "\n".join(lines).strip()
+
+    try:
+        decoded = json.loads(content)
+    except json.JSONDecodeError as exc:
+        raise ModelActionError(f"model_parse_error: {exc.msg}") from exc
+
+    if not isinstance(decoded, dict):
+        raise ModelActionError("model_parse_error: response is not a JSON object")
+    return decoded
+
+
+def parse_model_action(text: str) -> CyberAnalystAction:
+    payload = extract_json_object(text)
+    tool_name = payload.get("tool_name")
+    args = payload.get("args", {})
+
+    if not isinstance(tool_name, str) or not tool_name:
+        raise ModelActionError("model_parse_error: missing tool_name")
+    if not isinstance(args, dict):
+        raise ModelActionError("model_parse_error: args must be an object")
+
+    return CyberAnalystAction(tool_name=tool_name, args=args)
+
+
+def get_model_action(
+    client: OpenAI,
+    llm_config: LLMConfig,
+    task_id: str,
+    step: int,
+    obs: CyberAnalystObservation,
+) -> CyberAnalystAction:
+    try:
+        completion = client.chat.completions.create(
+            model=llm_config.model_name,
+            messages=[
+                {"role": "system", "content": SYSTEM_PROMPT},
+                {"role": "user", "content": build_user_prompt(task_id, step, obs)},
+            ],
+            temperature=llm_config.temperature,
+            max_tokens=llm_config.max_tokens,
+            stream=False,
+        )
+    except Exception as exc:
+        raise ModelActionError(f"model_request_error: {exc}") from exc
+
+    text = (completion.choices[0].message.content or "").strip()
+    if not text:
+        raise ModelActionError("model_parse_error: empty response")
+    return parse_model_action(text)
+
+
+def error_action(error: Exception) -> CyberAnalystAction:
+    message = single_line(str(error))
+    if message.startswith("model_request_error"):
+        tool_name = "model_request_error"
+    elif message.startswith("model_parse_error"):
+        tool_name = "model_parse_error"
+    else:
+        tool_name = "model_action_error"
+    return CyberAnalystAction(
+        tool_name=tool_name,
+        args={"message": message[:500]},
+    )
+
+
+def run_task(task_id: str, client: OpenAI, llm_config: LLMConfig) -> None:
+    log_start(task_id, llm_config)
+    rewards: list[float] = []
+    steps_taken = 0
+    final_score = 0.01
+    success = False
+
+    try:
+        with CyberAnalystEnv(base_url=ENV_URL).sync() as env:
+            reset_result = env.reset(task_id=task_id, seed=SEED)
+            obs = reset_result.observation
+
+            for step in range(1, MAX_STEPS + 1):
+                if obs.done:
+                    break
+
+                model_failed = False
+                try:
+                    action = get_model_action(client, llm_config, task_id, step, obs)
+                except ModelActionError as exc:
+                    action = error_action(exc)
+                    model_failed = True
+
+                result = env.step(action)
+                obs = result.observation
+                reward = float(result.reward or 0.0)
+                rewards.append(reward)
+                steps_taken = step
+
+                log_step(step, action, result.reward, result.done, obs.error)
+
+                if isinstance(obs.tool_result, dict) and "score" in obs.tool_result:
+                    final_score = float(obs.tool_result["score"])
+
+                if result.done or model_failed:
+                    success = final_score > 0.5
+                    break
+
+    except Exception as exc:
+        action = CyberAnalystAction(
+            tool_name="runtime_error",
+            args={"message": single_line(str(exc))[:500]},
+        )
+        steps_taken = max(steps_taken, 1)
+        rewards.append(0.01)
+        log_step(steps_taken, action, 0.01, True, single_line(str(exc)))
+
+    log_end(task_id, success, steps_taken, final_score, rewards)
+
+
+def selected_task_ids() -> list[str]:
+    task_override = os.getenv("TASK_NAME")
+    return [task_override] if task_override else TASK_IDS
+
+
+def main() -> None:
+    llm_config = build_llm_config()
+    try:
+        client = build_openai_client()
+    except KeyError:
+        print("HF_TOKEN must be set for inference.", file=sys.stderr, flush=True)
+        raise SystemExit(2) from None
+
+    for task_id in selected_task_ids():
+        run_task(task_id, client, llm_config)
+
+
+if __name__ == "__main__":
+    main()

openenv_Cyber_analyst.egg-info/PKG-INFO

@@ -0,0 +1,10 @@
+Metadata-Version: 2.4
+Name: openenv-Cyber_analyst
+Version: 0.1.0
+Summary: Cyber Analyst environment for OpenEnv
+Requires-Python: >=3.10
+Requires-Dist: openenv-core[core]>=0.2.2
+Requires-Dist: openai>=1.0.0
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0.0; extra == "dev"

openenv_Cyber_analyst.egg-info/SOURCES.txt

@@ -0,0 +1,22 @@
+README.md
+__init__.py
+client.py
+inference.py
+models.py
+pyproject.toml
+./__init__.py
+./client.py
+./inference.py
+./models.py
+openenv_Cyber_analyst.egg-info/PKG-INFO
+openenv_Cyber_analyst.egg-info/SOURCES.txt
+openenv_Cyber_analyst.egg-info/dependency_links.txt
+openenv_Cyber_analyst.egg-info/entry_points.txt
+openenv_Cyber_analyst.egg-info/requires.txt
+openenv_Cyber_analyst.egg-info/top_level.txt
+server/Cyber_analyst_environment.py
+server/__init__.py
+server/app.py
+server/graders.py
+server/tasks.py
+tests/test_environment.py

openenv_Cyber_analyst.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

openenv_Cyber_analyst.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+server = Cyber_analyst.server.app:main

openenv_Cyber_analyst.egg-info/requires.txt

@@ -0,0 +1,6 @@
+openenv-core[core]>=0.2.2
+openai>=1.0.0
+
+[dev]
+pytest>=8.0.0
+pytest-cov>=4.0.0

openenv_Cyber_analyst.egg-info/top_level.txt

@@ -0,0 +1 @@
+Cyber_analyst

sample_inference_script.py

@@ -0,0 +1,188 @@
+"""
+Inference Script Example
+===================================
+MANDATORY
+- Before submitting, ensure the following variables are defined in your environment configuration:
+    API_BASE_URL   The API endpoint for the LLM.
+    MODEL_NAME     The model identifier to use for inference.
+    HF_TOKEN       Your Hugging Face / API key.
+    LOCAL_IMAGE_NAME The name of the local image to use for the environment if you are using from_docker_image()
+                     method
+
+- Defaults are set only for API_BASE_URL and MODEL_NAME 
+    (and should reflect your active inference setup):
+    API_BASE_URL = os.getenv("API_BASE_URL", "<your-active-endpoint>")
+    MODEL_NAME = os.getenv("MODEL_NAME", "<your-active-model>")
+    
+- The inference script must be named `inference.py` and placed in the root directory of the project
+- Participants must use OpenAI Client for all LLM calls using above variables
+
+STDOUT FORMAT
+- The script must emit exactly three line types to stdout, in this order:
+
+    [START] task=<task_name> env=<benchmark> model=<model_name>
+    [STEP]  step=<n> action=<action_str> reward=<0.00> done=<true|false> error=<msg|null>
+    [END]   success=<true|false> steps=<n> score=<score> rewards=<r1,r2,...,rn>
+
+  Rules:
+    - One [START] line at episode begin.
+    - One [STEP] line per step, immediately after env.step() returns.
+    - One [END] line after env.close(), always emitted (even on exception).
+    - reward and rewards are formatted to 2 decimal places.
+    - done and success are lowercase booleans: true or false.
+    - error is the raw last_action_error string, or null if none.
+    - All fields on a single line with no newlines within a line.
+    - Each tasks should return score in [0, 1]
+
+  Example:
+    [START] task=click-test env=miniwob model=Qwen3-VL-30B
+    [STEP] step=1 action=click('123') reward=0.00 done=false error=null
+    [STEP] step=2 action=fill('456','text') reward=0.00 done=false error=null
+    [STEP] step=3 action=click('789') reward=1.00 done=true error=null
+    [END] success=true steps=3 score=1.00 rewards=0.00,0.00,1.00
+"""
+
+import asyncio
+import os
+import textwrap
+from typing import List, Optional
+
+from openai import OpenAI
+
+from my_env_v4 import MyEnvV4Action, MyEnvV4Env
+IMAGE_NAME = os.getenv("IMAGE_NAME") # If you are using docker image 
+API_KEY = os.getenv("HF_TOKEN") or os.getenv("API_KEY")
+
+API_BASE_URL = os.getenv("API_BASE_URL") or "https://router.huggingface.co/v1"
+MODEL_NAME = os.getenv("MODEL_NAME") or "Qwen/Qwen2.5-72B-Instruct"
+TASK_NAME = os.getenv("MY_ENV_V4_TASK", "echo")
+BENCHMARK = os.getenv("MY_ENV_V4_BENCHMARK", "my_env_v4")
+MAX_STEPS = 8
+TEMPERATURE = 0.7
+MAX_TOKENS = 150
+SUCCESS_SCORE_THRESHOLD = 0.1  # normalized score in [0, 1]
+
+# Max possible reward: each token contributes 0.1, across all steps
+_MAX_REWARD_PER_STEP = MAX_TOKENS * 0.1
+MAX_TOTAL_REWARD = MAX_STEPS * _MAX_REWARD_PER_STEP
+
+SYSTEM_PROMPT = textwrap.dedent(
+    """
+    You are interacting with a simple echo environment.
+    Each turn you must send a message. The environment will echo it back.
+    Reward is proportional to message length: reward = len(message) * 0.1
+    Your goal is to maximize total reward by sending meaningful, substantive messages.
+    Reply with exactly one message string — no quotes, no prefixes, just the message text.
+    """
+).strip()
+
+
+def log_start(task: str, env: str, model: str) -> None:
+    print(f"[START] task={task} env={env} model={model}", flush=True)
+
+
+def log_step(step: int, action: str, reward: float, done: bool, error: Optional[str]) -> None:
+    error_val = error if error else "null"
+    done_val = str(done).lower()
+    print(
+        f"[STEP] step={step} action={action} reward={reward:.2f} done={done_val} error={error_val}",
+        flush=True,
+    )
+
+
+def log_end(success: bool, steps: int, score: float, rewards: List[float]) -> None:
+    rewards_str = ",".join(f"{r:.2f}" for r in rewards)
+    print(f"[END] success={str(success).lower()} steps={steps} score={score:.3f} rewards={rewards_str}", flush=True)
+
+
+def build_user_prompt(step: int, last_echoed: str, last_reward: float, history: List[str]) -> str:
+    history_block = "\n".join(history[-4:]) if history else "None"
+    return textwrap.dedent(
+        f"""
+        Step: {step}
+        Last echoed message: {last_echoed!r}
+        Last reward: {last_reward:.2f}
+        Previous steps:
+        {history_block}
+        Send your next message.
+        """
+    ).strip()
+
+
+def get_model_message(client: OpenAI, step: int, last_echoed: str, last_reward: float, history: List[str]) -> str:
+    user_prompt = build_user_prompt(step, last_echoed, last_reward, history)
+    try:
+        completion = client.chat.completions.create(
+            model=MODEL_NAME,
+            messages=[
+                {"role": "system", "content": SYSTEM_PROMPT},
+                {"role": "user", "content": user_prompt},
+            ],
+            temperature=TEMPERATURE,
+            max_tokens=MAX_TOKENS,
+            stream=False,
+        )
+        text = (completion.choices[0].message.content or "").strip()
+        return text if text else "hello"
+    except Exception as exc:
+        print(f"[DEBUG] Model request failed: {exc}", flush=True)
+        return "hello"
+
+
+async def main() -> None:
+    client = OpenAI(base_url=API_BASE_URL, api_key=API_KEY)
+
+    env = await MyEnvV4Env.from_docker_image(IMAGE_NAME)
+
+    history: List[str] = []
+    rewards: List[float] = []
+    steps_taken = 0
+    score = 0.0
+    success = False
+
+    log_start(task=TASK_NAME, env=BENCHMARK, model=MODEL_NAME)
+
+    try:
+        result = await env.reset() # OpenENV.reset()
+        last_echoed = result.observation.echoed_message
+        last_reward = 0.0
+
+        for step in range(1, MAX_STEPS + 1):
+            if result.done:
+                break
+
+            message = get_model_message(client, step, last_echoed, last_reward, history)
+
+            result = await env.step(MyEnvV4Action(message=message))
+            obs = result.observation
+
+            reward = result.reward or 0.0
+            done = result.done
+            error = None
+
+            rewards.append(reward)
+            steps_taken = step
+            last_echoed = obs.echoed_message
+            last_reward = reward
+
+            log_step(step=step, action=message, reward=reward, done=done, error=error)
+
+            history.append(f"Step {step}: {message!r} -> reward {reward:+.2f}")
+
+            if done:
+                break
+
+        score = sum(rewards) / MAX_TOTAL_REWARD if MAX_TOTAL_REWARD > 0 else 0.0
+        score = min(max(score, 0.0), 1.0)  # clamp to [0, 1]
+        success = score >= SUCCESS_SCORE_THRESHOLD
+
+    finally:
+        try:
+            await env.close()
+        except Exception as e:
+            print(f"[DEBUG] env.close() error (container cleanup): {e}", flush=True)
+        log_end(success=success, steps=steps_taken, score=score, rewards=rewards)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

scripts/validate-submission.sh

@@ -0,0 +1,188 @@
+#!/usr/bin/env bash
+#
+# validate-submission.sh — OpenEnv Submission Validator
+#
+# Checks that your HF Space is live, Docker image builds, and openenv validate passes.
+#
+# Prerequisites:
+#   - Docker:       https://docs.docker.com/get-docker/
+#   - openenv-core: pip install openenv-core
+#   - curl (usually pre-installed)
+#
+# Run:
+#   curl -fsSL https://raw.githubusercontent.com/<owner>/<repo>/main/scripts/validate-submission.sh | bash -s -- <ping_url> [repo_dir]
+#
+#   Or download and run locally:
+#     chmod +x validate-submission.sh
+#     ./validate-submission.sh <ping_url> [repo_dir]
+#
+# Arguments:
+#   ping_url   Your HuggingFace Space URL (e.g. https://your-space.hf.space)
+#   repo_dir   Path to your repo (default: current directory)
+#
+# Examples:
+#   ./validate-submission.sh https://my-team.hf.space
+#   ./validate-submission.sh https://my-team.hf.space ./my-repo
+#
+
+set -uo pipefail
+
+DOCKER_BUILD_TIMEOUT=600
+if [ -t 1 ]; then
+  RED='\033[0;31m'
+  GREEN='\033[0;32m'
+  YELLOW='\033[1;33m'
+  BOLD='\033[1m'
+  NC='\033[0m'
+else
+  RED='' GREEN='' YELLOW='' BOLD='' NC=''
+fi
+
+run_with_timeout() {
+  local secs="$1"; shift
+  if command -v timeout &>/dev/null; then
+    timeout "$secs" "$@"
+  elif command -v gtimeout &>/dev/null; then
+    gtimeout "$secs" "$@"
+  else
+    "$@" &
+    local pid=$!
+    ( sleep "$secs" && kill "$pid" 2>/dev/null ) &
+    local watcher=$!
+    wait "$pid" 2>/dev/null
+    local rc=$?
+    kill "$watcher" 2>/dev/null
+    wait "$watcher" 2>/dev/null
+    return $rc
+  fi
+}
+
+portable_mktemp() {
+  local prefix="${1:-validate}"
+  mktemp "${TMPDIR:-/tmp}/${prefix}-XXXXXX" 2>/dev/null || mktemp
+}
+
+CLEANUP_FILES=()
+cleanup() { rm -f "${CLEANUP_FILES[@]+"${CLEANUP_FILES[@]}"}"; }
+trap cleanup EXIT
+
+PING_URL="${1:-}"
+REPO_DIR="${2:-.}"
+
+if [ -z "$PING_URL" ]; then
+  printf "Usage: %s <ping_url> [repo_dir]\n" "$0"
+  printf "\n"
+  printf "  ping_url   Your HuggingFace Space URL (e.g. https://your-space.hf.space)\n"
+  printf "  repo_dir   Path to your repo (default: current directory)\n"
+  exit 1
+fi
+
+if ! REPO_DIR="$(cd "$REPO_DIR" 2>/dev/null && pwd)"; then
+  printf "Error: directory '%s' not found\n" "${2:-.}"
+  exit 1
+fi
+PING_URL="${PING_URL%/}"
+export PING_URL
+PASS=0
+
+log()  { printf "[%s] %b\n" "$(date -u +%H:%M:%S)" "$*"; }
+pass() { log "${GREEN}PASSED${NC} -- $1"; PASS=$((PASS + 1)); }
+fail() { log "${RED}FAILED${NC} -- $1"; }
+hint() { printf "  ${YELLOW}Hint:${NC} %b\n" "$1"; }
+stop_at() {
+  printf "\n"
+  printf "${RED}${BOLD}Validation stopped at %s.${NC} Fix the above before continuing.\n" "$1"
+  exit 1
+}
+
+printf "\n"
+printf "${BOLD}========================================${NC}\n"
+printf "${BOLD}  OpenEnv Submission Validator${NC}\n"
+printf "${BOLD}========================================${NC}\n"
+log "Repo:     $REPO_DIR"
+log "Ping URL: $PING_URL"
+printf "\n"
+
+log "${BOLD}Step 1/3: Pinging HF Space${NC} ($PING_URL/reset) ..."
+
+CURL_OUTPUT=$(portable_mktemp "validate-curl")
+CLEANUP_FILES+=("$CURL_OUTPUT")
+HTTP_CODE=$(curl -s -o "$CURL_OUTPUT" -w "%{http_code}" -X POST \
+  -H "Content-Type: application/json" -d '{}' \
+  "$PING_URL/reset" --max-time 30 2>"$CURL_OUTPUT" || printf "000")
+
+if [ "$HTTP_CODE" = "200" ]; then
+  pass "HF Space is live and responds to /reset"
+elif [ "$HTTP_CODE" = "000" ]; then
+  fail "HF Space not reachable (connection failed or timed out)"
+  hint "Check your network connection and that the Space is running."
+  hint "Try: curl -s -o /dev/null -w '%%{http_code}' -X POST $PING_URL/reset"
+  stop_at "Step 1"
+else
+  fail "HF Space /reset returned HTTP $HTTP_CODE (expected 200)"
+  hint "Make sure your Space is running and the URL is correct."
+  hint "Try opening $PING_URL in your browser first."
+  stop_at "Step 1"
+fi
+
+log "${BOLD}Step 2/3: Running docker build${NC} ..."
+
+if ! command -v docker &>/dev/null; then
+  fail "docker command not found"
+  hint "Install Docker: https://docs.docker.com/get-docker/"
+  stop_at "Step 2"
+fi
+
+if [ -f "$REPO_DIR/Dockerfile" ]; then
+  DOCKER_CONTEXT="$REPO_DIR"
+  DOCKERFILE="$REPO_DIR/Dockerfile"
+elif [ -f "$REPO_DIR/server/Dockerfile" ]; then
+  DOCKER_CONTEXT="$REPO_DIR"
+  DOCKERFILE="$REPO_DIR/server/Dockerfile"
+else
+  fail "No Dockerfile found in repo root or server/ directory"
+  stop_at "Step 2"
+fi
+
+log "  Found Dockerfile at $DOCKERFILE"
+log "  Docker context: $DOCKER_CONTEXT"
+
+BUILD_OK=false
+BUILD_OUTPUT=$(run_with_timeout "$DOCKER_BUILD_TIMEOUT" docker build -f "$DOCKERFILE" "$DOCKER_CONTEXT" 2>&1) && BUILD_OK=true
+
+if [ "$BUILD_OK" = true ]; then
+  pass "Docker build succeeded"
+else
+  fail "Docker build failed (timeout=${DOCKER_BUILD_TIMEOUT}s)"
+  printf "%s\n" "$BUILD_OUTPUT" | tail -20
+  stop_at "Step 2"
+fi
+
+log "${BOLD}Step 3/3: Running openenv validate${NC} ..."
+
+if ! command -v openenv &>/dev/null; then
+  fail "openenv command not found"
+  hint "Install it: pip install openenv-core"
+  stop_at "Step 3"
+fi
+
+VALIDATE_OK=false
+VALIDATE_OUTPUT=$(cd "$REPO_DIR" && openenv validate 2>&1) && VALIDATE_OK=true
+
+if [ "$VALIDATE_OK" = true ]; then
+  pass "openenv validate passed"
+  [ -n "$VALIDATE_OUTPUT" ] && log "  $VALIDATE_OUTPUT"
+else
+  fail "openenv validate failed"
+  printf "%s\n" "$VALIDATE_OUTPUT"
+  stop_at "Step 3"
+fi
+
+printf "\n"
+printf "${BOLD}========================================${NC}\n"
+printf "${GREEN}${BOLD}  All 3/3 checks passed!${NC}\n"
+printf "${GREEN}${BOLD}  Your submission is ready to submit.${NC}\n"
+printf "${BOLD}========================================${NC}\n"
+printf "\n"
+
+exit 0

server/Cyber_analyst_environment.py

@@ -0,0 +1,506 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+"""SecOps Evidence Gym environment implementation."""
+
+from __future__ import annotations
+
+import hashlib
+import json
+from collections import Counter
+from typing import Any
+from uuid import uuid4
+
+from openenv.core.env_server.interfaces import Environment
+
+try:
+    from ..models import (
+        CyberAnalystAction,
+        CyberAnalystObservation,
+        CyberAnalystState,
+    )
+    from .graders import safe_reward, score_report
+    from .tasks import DEFAULT_TASK_ID, TOOL_CATALOG, build_scenario
+except ImportError:  # pragma: no cover - supports direct module execution
+    from models import CyberAnalystAction, CyberAnalystObservation, CyberAnalystState
+    from server.graders import safe_reward, score_report
+    from server.tasks import DEFAULT_TASK_ID, TOOL_CATALOG, build_scenario
+
+
+class CyberAnalystEnvironment(
+    Environment[CyberAnalystAction, CyberAnalystObservation, CyberAnalystState]
+):
+    """A safe, deterministic evidence-grounded cyber analyst benchmark."""
+
+    SUPPORTS_CONCURRENT_SESSIONS: bool = True
+    MAX_STEPS = 12
+    REPEAT_HARD_STOP = 6
+
+    def __init__(self):
+        super().__init__()
+        self._scenario: dict[str, Any] = {}
+        self._state = CyberAnalystState()
+        self._discovered_evidence: set[str] = set()
+        self._candidate_findings: dict[str, dict[str, Any]] = {}
+        self._verified_findings: list[dict[str, Any]] = []
+        self._validated_finding_ids: set[str] = set()
+        self._action_counts: Counter[str] = Counter()
+        self._last_score_breakdown: dict[str, Any] = {}
+        self._trajectory_events: list[dict[str, Any]] = []
+        self._initialize_episode(DEFAULT_TASK_ID, seed=None, episode_id=None)
+
+    def reset(
+        self,
+        seed: int | None = None,
+        episode_id: str | None = None,
+        task_id: str = DEFAULT_TASK_ID,
+        **_: Any,
+    ) -> CyberAnalystObservation:
+        """Reset the selected deterministic task."""
+
+        self._initialize_episode(task_id=task_id, seed=seed, episode_id=episode_id)
+        tool_result = {
+            "message": "Cyber Analyst environment ready.",
+            "allowed_scope": "Synthetic artifacts only. No live targets or shell.",
+        }
+        obs = self._observation(
+            tool_result={
+                **tool_result,
+                "trajectory_jsonl": self.export_trajectory_jsonl(),
+            },
+            reward=0.01,
+        )
+        self._record_trajectory("reset", None, tool_result, obs.reward, obs.done, obs.error)
+        return obs
+
+    def step(  # type: ignore[override]
+        self,
+        action: CyberAnalystAction,
+        timeout_s: float | None = None,
+        **_: Any,
+    ) -> CyberAnalystObservation:
+        """Execute one bounded simulator tool call."""
+
+        del timeout_s
+
+        if self._state.done:
+            tool_result = {"message": "Episode is already complete."}
+            obs = self._observation(
+                tool_result=tool_result,
+                reward=0.01,
+                done=True,
+                error="episode_already_done",
+            )
+            self._record_trajectory("step", action, tool_result, obs.reward, obs.done, obs.error)
+            return obs
+
+        self._state.step_count += 1
+        self._state.step_budget_remaining = max(
+            0, self.MAX_STEPS - self._state.step_count
+        )
+
+        signature = self._action_signature(action)
+        self._action_counts[signature] += 1
+        repeat_count = self._action_counts[signature]
+
+        if repeat_count >= self.REPEAT_HARD_STOP:
+            self._state.phase = "done"
+            self._state.done = True
+            self._last_score_breakdown = {
+                "score": 0.03,
+                "repeat_hard_stop": True,
+                "signature": signature,
+            }
+            tool_result = {"message": "Episode stopped after repeated identical actions."}
+            obs = self._observation(
+                tool_result=tool_result,
+                reward=0.03,
+                done=True,
+                error="repeat_hard_stop",
+            )
+            self._record_trajectory("step", action, tool_result, obs.reward, obs.done, obs.error)
+            return obs
+
+        handler = getattr(self, f"_tool_{action.tool_name}", None)
+        if handler is None:
+            tool_result = {
+                "ok": False,
+                "message": f"Unsupported tool: {action.tool_name}",
+                "available_tools": [tool["name"] for tool in TOOL_CATALOG],
+            }
+            obs = self._step_observation(
+                tool_result=tool_result,
+                repeat_count=repeat_count,
+                error="unsupported_tool",
+            )
+            self._record_trajectory("step", action, tool_result, obs.reward, obs.done, obs.error)
+            return obs
+
+        try:
+            result, reward_delta, done = handler(action.args)
+            error = ""
+        except Exception as exc:  # pragma: no cover - defensive rollout guard
+            result = {"ok": False, "message": str(exc)}
+            reward_delta = -0.05
+            done = False
+            error = exc.__class__.__name__
+
+        if self._state.step_budget_remaining <= 0 and not done:
+            done = True
+            self._state.phase = "done"
+            self._state.done = True
+            result = {
+                **result,
+                "timeout": True,
+                "message": "Step budget exhausted before report submission.",
+            }
+            reward_delta -= 0.10
+
+        obs = self._step_observation(
+            tool_result=result,
+            repeat_count=repeat_count,
+            reward_delta=reward_delta,
+            done=done,
+            error=error,
+        )
+        self._record_trajectory("step", action, result, obs.reward, obs.done, obs.error)
+        return obs
+
+    @property
+    def state(self) -> CyberAnalystState:
+        """Return the current episode state summary."""
+
+        return self._state
+
+    def _initialize_episode(
+        self, task_id: str, seed: int | None, episode_id: str | None
+    ) -> None:
+        self._scenario = build_scenario(task_id, seed)
+        self._discovered_evidence = set()
+        self._candidate_findings = {}
+        self._verified_findings = []
+        self._validated_finding_ids = set()
+        self._action_counts = Counter()
+        self._last_score_breakdown = {}
+        self._trajectory_events = []
+        self._state = CyberAnalystState(
+            episode_id=episode_id or str(uuid4()),
+            step_count=0,
+            task_id=self._scenario["task_id"],
+            seed=seed,
+            phase="investigate",
+            step_budget_remaining=self.MAX_STEPS,
+            recent_evidence_ids=[],
+            verified_finding_ids=[],
+            done=False,
+        )
+
+    def export_trajectory_jsonl(self) -> str:
+        """Return the current episode trajectory as JSONL for offline analysis."""
+
+        return "\n".join(
+            json.dumps(event, sort_keys=True, default=str)
+            for event in self._trajectory_events
+        )
+
+    def _record_trajectory(
+        self,
+        event_type: str,
+        action: CyberAnalystAction | None,
+        tool_result: dict[str, Any],
+        reward: float | int | None,
+        done: bool,
+        error: str,
+    ) -> None:
+        action_payload = None
+        if action is not None:
+            action_payload = action.model_dump(exclude_none=True)
+        self._trajectory_events.append(
+            {
+                "episode_id": self._state.episode_id,
+                "task_id": self._state.task_id,
+                "seed": self._state.seed,
+                "event_type": event_type,
+                "step": self._state.step_count,
+                "phase": self._state.phase,
+                "action": action_payload,
+                "tool_result": tool_result,
+                "evidence_ids": sorted(self._discovered_evidence),
+                "verified_finding_ids": list(self._state.verified_finding_ids),
+                "reward": reward,
+                "done": done,
+                "error": error,
+            }
+        )
+
+    def _observation(
+        self,
+        tool_result: dict[str, Any] | None = None,
+        reward: float = 0.01,
+        done: bool | None = None,
+        error: str = "",
+    ) -> CyberAnalystObservation:
+        done_value = self._state.done if done is None else done
+        return CyberAnalystObservation(
+            task_id=self._scenario.get("task_id", ""),
+            alert=self._scenario.get("alert", ""),
+            phase=self._state.phase,
+            tool_catalog=TOOL_CATALOG,
+            tool_result=tool_result or {},
+            evidence_ids=sorted(self._discovered_evidence),
+            verified_findings=list(self._verified_findings),
+            candidate_findings=list(self._candidate_findings.values()),
+            step_budget_remaining=self._state.step_budget_remaining,
+            score_breakdown=dict(self._last_score_breakdown),
+            error=error,
+            done=done_value,
+            reward=safe_reward(reward),
+        )
+
+    def _step_observation(
+        self,
+        tool_result: dict[str, Any],
+        repeat_count: int,
+        reward_delta: float = 0.0,
+        done: bool = False,
+        error: str = "",
+    ) -> CyberAnalystObservation:
+        reward = 0.04 + reward_delta - 0.01
+        if repeat_count > 2:
+            reward -= 0.03 * (repeat_count - 2)
+
+        if done:
+            self._state.phase = "done"
+            self._state.done = True
+
+        self._state.recent_evidence_ids = sorted(self._discovered_evidence)[-5:]
+        self._state.verified_finding_ids = [
+            finding["finding_id"] for finding in self._verified_findings
+        ]
+
+        return self._observation(
+            tool_result=tool_result,
+            reward=safe_reward(reward),
+            done=self._state.done,
+            error=error,
+        )
+
+    def _action_signature(self, action: CyberAnalystAction) -> str:
+        payload = {
+            "tool_name": action.tool_name,
+            "args": action.args,
+        }
+        encoded = json.dumps(payload, sort_keys=True, default=str)
+        return hashlib.sha256(encoded.encode("utf-8")).hexdigest()[:16]
+
+    def _record_evidence(self, evidence_ids: list[str]) -> int:
+        relevant = set(self._scenario.get("required_evidence", [])) | set(
+            self._scenario.get("supporting_evidence", [])
+        )
+        new_relevant = 0
+        for evidence_id in evidence_ids:
+            if evidence_id not in self._discovered_evidence and evidence_id in relevant:
+                new_relevant += 1
+            self._discovered_evidence.add(evidence_id)
+        return new_relevant
+
+    def _filter_entries(
+        self, entries: list[dict[str, Any]], service_id: str = "", query: str = ""
+    ) -> list[dict[str, Any]]:
+        normalized_service = self._resolve_service_id(service_id).lower()
+        normalized_query = query.strip().lower()
+        matches: list[dict[str, Any]] = []
+        for entry in entries:
+            service_matches = (
+                not normalized_service
+                or str(entry.get("service_id", "")).lower() == normalized_service
+            )
+            search_blob = " ".join(
+                [
+                    str(entry.get("text", "")),
+                    str(entry.get("source", "")),
+                    " ".join(str(tag) for tag in entry.get("tags", [])),
+                ]
+            ).lower()
+            query_matches = not normalized_query or normalized_query in search_blob
+            if service_matches and query_matches:
+                matches.append(entry)
+        return matches
+
+    def _resolve_service_id(self, service_id: str) -> str:
+        normalized = service_id.strip()
+        aliases = self._scenario.get("service_aliases", {})
+        return str(aliases.get(normalized, normalized))
+
+    def _evidence_payload(self, entries: list[dict[str, Any]]) -> dict[str, Any]:
+        evidence_ids = [entry["evidence_id"] for entry in entries]
+        new_relevant = self._record_evidence(evidence_ids)
+        return {
+            "ok": True,
+            "evidence_ids": evidence_ids,
+            "new_relevant_evidence": new_relevant,
+            "entries": [
+                {
+                    "evidence_id": entry["evidence_id"],
+                    "service_id": entry.get("service_id", ""),
+                    "source": entry.get("source", ""),
+                    "text": entry.get("text", ""),
+                }
+                for entry in entries
+            ],
+        }
+
+    def _tool_list_assets(self, args: dict[str, Any]) -> tuple[dict[str, Any], float, bool]:
+        del args
+        return {"ok": True, "assets": self._scenario["assets"]}, 0.0, False
+
+    def _tool_get_log_events(
+        self, args: dict[str, Any]
+    ) -> tuple[dict[str, Any], float, bool]:
+        entries = self._filter_entries(
+            self._scenario.get("logs", []),
+            service_id=str(args.get("service_id", "")),
+            query=str(args.get("query", "")),
+        )
+        payload = self._evidence_payload(entries)
+        return payload, 0.02 * payload["new_relevant_evidence"], False
+
+    def _tool_check_security_headers(
+        self, args: dict[str, Any]
+    ) -> tuple[dict[str, Any], float, bool]:
+        requested_service = self._resolve_service_id(str(args.get("service_id", ""))).lower()
+        snapshots = self._scenario.get("headers", {})
+        results = []
+        evidence_ids = []
+        for service_id, snapshot in snapshots.items():
+            if requested_service and service_id.lower() != requested_service:
+                continue
+            evidence_ids.append(snapshot["evidence_id"])
+            results.append(
+                {
+                    "service_id": service_id,
+                    "evidence_id": snapshot["evidence_id"],
+                    "present": snapshot.get("present", []),
+                    "missing": snapshot.get("missing", []),
+                    "passed": not snapshot.get("missing"),
+                }
+            )
+        new_relevant = self._record_evidence(evidence_ids)
+        return (
+            {
+                "ok": True,
+                "evidence_ids": evidence_ids,
+                "new_relevant_evidence": new_relevant,
+                "header_results": results,
+            },
+            0.02 * new_relevant,
+            False,
+        )
+
+    def _tool_search_repo(self, args: dict[str, Any]) -> tuple[dict[str, Any], float, bool]:
+        entries = self._filter_entries(
+            self._scenario.get("repo", []), query=str(args.get("query", ""))
+        )
+        payload = self._evidence_payload(entries)
+        return payload, 0.02 * payload["new_relevant_evidence"], False
+
+    def _tool_scan_dependencies(
+        self, args: dict[str, Any]
+    ) -> tuple[dict[str, Any], float, bool]:
+        del args
+        payload = self._evidence_payload(self._scenario.get("dependencies", []))
+        return payload, 0.02 * payload["new_relevant_evidence"], False
+
+    def _tool_create_finding(
+        self, args: dict[str, Any]
+    ) -> tuple[dict[str, Any], float, bool]:
+        evidence_ids = args.get("evidence_ids", [])
+        if isinstance(evidence_ids, str):
+            evidence_ids = [evidence_ids]
+        evidence_ids = [str(evidence_id) for evidence_id in evidence_ids]
+
+        finding_id = f"FND-{len(self._candidate_findings) + 1:03d}"
+        finding = {
+            "finding_id": finding_id,
+            "finding_type": str(args.get("finding_type", "")),
+            "evidence_ids": evidence_ids,
+            "severity_guess": str(args.get("severity_guess", "")),
+            "remediation": str(args.get("remediation", "")),
+            "validated": False,
+            "matching_gt_id": None,
+        }
+        self._candidate_findings[finding_id] = finding
+
+        well_formed = bool(
+            finding["finding_type"] and evidence_ids and finding["remediation"]
+        )
+        return (
+            {"ok": True, "finding_id": finding_id, "finding": finding},
+            0.03 if well_formed else 0.0,
+            False,
+        )
+
+    def _tool_validate_finding(
+        self, args: dict[str, Any]
+    ) -> tuple[dict[str, Any], float, bool]:
+        finding_id = str(args.get("finding_id", ""))
+        finding = self._candidate_findings.get(finding_id)
+        if finding is None:
+            return (
+                {"ok": False, "message": f"Unknown finding_id: {finding_id}"},
+                -0.03,
+                False,
+            )
+
+        expected_type = self._scenario["finding_type"]
+        required_evidence = set(self._scenario.get("required_evidence", []))
+        supplied_evidence = set(finding.get("evidence_ids", []))
+        verified = (
+            finding.get("finding_type") == expected_type
+            and bool(required_evidence & supplied_evidence)
+        )
+        self._validated_finding_ids.add(finding_id)
+        finding["validated"] = verified
+        finding["matching_gt_id"] = self._scenario["ground_truth_id"] if verified else None
+
+        if verified and not any(
+            item["finding_id"] == finding_id for item in self._verified_findings
+        ):
+            self._verified_findings.append(dict(finding))
+
+        return (
+            {
+                "ok": True,
+                "finding_id": finding_id,
+                "verified": verified,
+                "matching_gt_id": finding["matching_gt_id"],
+            },
+            0.08 if verified else -0.02,
+            False,
+        )
+
+    def _tool_submit_report(
+        self, args: dict[str, Any]
+    ) -> tuple[dict[str, Any], float, bool]:
+        report = args.get("report_json", {})
+        score, breakdown = score_report(
+            self._scenario["task_id"],
+            report,
+            verified_findings=self._verified_findings,
+            validation_attempted=bool(self._validated_finding_ids),
+        )
+        self._last_score_breakdown = breakdown
+        return (
+            {
+                "ok": True,
+                "submitted": True,
+                "score": score,
+                "score_breakdown": breakdown,
+                "trajectory_jsonl": self.export_trajectory_jsonl(),
+            },
+            score,
+            True,
+        )

server/graders.py

@@ -0,0 +1,169 @@
+"""Deterministic graders for the Cyber Analyst OpenEnv tasks."""
+
+from __future__ import annotations
+
+import json
+from typing import Any
+
+try:
+    from .tasks import SCENARIOS
+except ImportError:  # pragma: no cover - supports direct module execution
+    from tasks import SCENARIOS
+
+
+MIN_SCORE = 0.01
+MAX_SCORE = 0.99
+
+
+def safe_reward(score: float | int | None) -> float:
+    """Clamp validator-facing scores to the strict open interval (0, 1)."""
+
+    try:
+        value = float(score if score is not None else 0.0)
+    except (TypeError, ValueError):
+        value = 0.0
+    return max(MIN_SCORE, min(MAX_SCORE, value))
+
+
+def _coerce_report(report: Any) -> dict[str, Any]:
+    if isinstance(report, dict):
+        return report
+    if isinstance(report, str):
+        try:
+            decoded = json.loads(report)
+        except json.JSONDecodeError:
+            return {"summary": report, "findings": []}
+        return decoded if isinstance(decoded, dict) else {"findings": []}
+    return {"findings": []}
+
+
+def _text_contains_any(text: str, keywords: list[str]) -> bool:
+    lowered = text.lower()
+    return any(keyword.lower() in lowered for keyword in keywords)
+
+
+def _report_findings(report: dict[str, Any]) -> list[dict[str, Any]]:
+    findings = report.get("findings", [])
+    if isinstance(findings, dict):
+        findings = [findings]
+    return [finding for finding in findings if isinstance(finding, dict)]
+
+
+def score_report(
+    task_id: str,
+    report: Any,
+    verified_findings: list[dict[str, Any]] | None = None,
+    validation_attempted: bool = False,
+) -> tuple[float, dict[str, Any]]:
+    """Score a submitted report against one task's deterministic ground truth."""
+
+    scenario = SCENARIOS.get(task_id)
+    report_dict = _coerce_report(report)
+    report_findings = _report_findings(report_dict)
+    verified_findings = verified_findings or []
+
+    if scenario is None:
+        return MIN_SCORE, {"unknown_task": task_id}
+
+    expected_type = scenario["finding_type"]
+    expected_evidence = set(scenario.get("required_evidence", [])) | set(
+        scenario.get("supporting_evidence", [])
+    )
+
+    matching_verified = [
+        finding
+        for finding in verified_findings
+        if finding.get("finding_type") == expected_type
+    ]
+    matching_report = [
+        finding for finding in report_findings if finding.get("finding_type") == expected_type
+    ]
+
+    score = 0.05
+    breakdown: dict[str, Any] = {
+        "base": 0.05,
+        "verified_correct": 0.0,
+        "valid_evidence": 0.0,
+        "actionable_remediation": 0.0,
+        "hallucination_penalty": 0.0,
+        "validation_penalty": 0.0,
+    }
+
+    if matching_verified and matching_report:
+        impact_text = " ".join(
+            str(finding.get("impact", "")) + " " + str(finding.get("description", ""))
+            for finding in matching_report
+        )
+        if _text_contains_any(impact_text, scenario.get("impact_keywords", [])):
+            score += 0.60
+            breakdown["verified_correct"] = 0.60
+
+    report_evidence: set[str] = set()
+    for finding in matching_report:
+        evidence_ids = finding.get("evidence_ids", [])
+        if isinstance(evidence_ids, str):
+            evidence_ids = [evidence_ids]
+        report_evidence.update(str(evidence_id) for evidence_id in evidence_ids)
+
+    if report_evidence & expected_evidence:
+        score += 0.15
+        breakdown["valid_evidence"] = 0.15
+
+    remediation_text = " ".join(
+        str(finding.get("remediation", "")) for finding in matching_report
+    )
+    if _text_contains_any(remediation_text, scenario.get("remediation_keywords", [])):
+        score += 0.15
+        breakdown["actionable_remediation"] = 0.15
+
+    verified_types = {finding.get("finding_type") for finding in verified_findings}
+    hallucinated = [
+        finding
+        for finding in report_findings
+        if finding.get("finding_type") not in verified_types
+    ]
+    if hallucinated:
+        penalty = 0.40 * len(hallucinated)
+        score -= penalty
+        breakdown["hallucination_penalty"] = -penalty
+
+    if not validation_attempted:
+        score -= 0.20
+        breakdown["validation_penalty"] = -0.20
+
+    final_score = safe_reward(score)
+    breakdown["raw_score"] = round(score, 4)
+    breakdown["score"] = final_score
+    return final_score, breakdown
+
+
+def _payload_from_args(*args: Any, **kwargs: Any) -> dict[str, Any]:
+    if args and isinstance(args[0], dict):
+        payload = dict(args[0])
+    else:
+        payload = {}
+    payload.update(kwargs)
+    return payload
+
+
+def grade_task(task_id: str, *args: Any, **kwargs: Any) -> float:
+    """Manifest-friendly grader adapter."""
+
+    payload = _payload_from_args(*args, **kwargs)
+    report = payload.get("report") or payload.get("report_json") or payload
+    verified_findings = payload.get("verified_findings", [])
+    validation_attempted = bool(payload.get("validation_attempted", False))
+    score, _ = score_report(task_id, report, verified_findings, validation_attempted)
+    return score
+
+
+def grade_secret_exposure_easy(*args: Any, **kwargs: Any) -> float:
+    return grade_task("secret_exposure_easy", *args, **kwargs)
+
+
+def grade_missing_security_headers_medium(*args: Any, **kwargs: Any) -> float:
+    return grade_task("missing_security_headers_medium", *args, **kwargs)
+
+
+def grade_authz_boundary_hard(*args: Any, **kwargs: Any) -> float:
+    return grade_task("authz_boundary_hard", *args, **kwargs)

server/tasks.py

@@ -0,0 +1,283 @@
+"""Deterministic scenario registry for SecOps Evidence Gym."""
+
+from __future__ import annotations
+
+from copy import deepcopy
+from random import Random
+from typing import Any
+
+
+DEFAULT_TASK_ID = "secret_exposure_easy"
+
+TOOL_CATALOG: list[dict[str, Any]] = [
+    {
+        "name": "list_assets",
+        "description": "List synthetic services, routes, and artifact collections.",
+        "args": {},
+    },
+    {
+        "name": "get_log_events",
+        "description": "Return sanitized telemetry evidence ids for a service/query.",
+        "args": {"service_id": "str", "query": "str"},
+    },
+    {
+        "name": "check_security_headers",
+        "description": "Inspect a service header snapshot and return pass/fail evidence.",
+        "args": {"service_id": "str"},
+    },
+    {
+        "name": "search_repo",
+        "description": "Search synthetic repo/config snippets for evidence ids.",
+        "args": {"query": "str"},
+    },
+    {
+        "name": "scan_dependencies",
+        "description": "Inspect a synthetic dependency manifest excerpt.",
+        "args": {},
+    },
+    {
+        "name": "create_finding",
+        "description": "Store a candidate finding for verifier review.",
+        "args": {
+            "finding_type": "str",
+            "evidence_ids": "list[str]",
+            "severity_guess": "str",
+            "remediation": "str",
+        },
+    },
+    {
+        "name": "validate_finding",
+        "description": "Run the deterministic verifier for a candidate finding.",
+        "args": {"finding_id": "str"},
+    },
+    {
+        "name": "submit_report",
+        "description": "Submit the final structured report and end the episode.",
+        "args": {"report_json": "dict"},
+    },
+]
+
+
+SCENARIOS: dict[str, dict[str, Any]] = {
+    "secret_exposure_easy": {
+        "task_id": "secret_exposure_easy",
+        "difficulty": "easy",
+        "title": "Secret exposure in repo snapshot",
+        "alert": (
+            "Repository hygiene alert: a synthetic config snapshot may contain "
+            "credential-like material. Investigate, cite evidence, validate, and "
+            "submit a remediation report."
+        ),
+        "ground_truth_id": "GT-SECRET-001",
+        "finding_type": "secret_exposure",
+        "severity": "high",
+        "required_evidence": ["EVID-101"],
+        "remediation_keywords": ["rotate", "remove"],
+        "impact_keywords": ["secret", "credential", "api key", "leak"],
+        "assets": {
+            "services": ["gateway", "profile-service", "admin-service"],
+            "routes": ["/login", "/profile", "/admin/export"],
+            "collections": ["repo_snapshot", "telemetry", "headers", "dependencies"],
+        },
+        "repo": [
+            {
+                "evidence_id": "EVID-101",
+                "source": "repo_snapshot/config/profile-service.env",
+                "service_id": "profile-service",
+                "tags": ["api key", "secret", "credential", "config"],
+                "text": (
+                    "PROFILE_EXPORT_API_KEY=CYBERLAB_DO_NOT_USE_API_KEY_7F3A "
+                    "# synthetic lab credential"
+                ),
+            },
+            {
+                "evidence_id": "EVID-102",
+                "source": "repo_snapshot/config/gateway.env",
+                "service_id": "gateway",
+                "tags": ["config", "benign"],
+                "text": "LOG_LEVEL=info; FEATURE_SAFE_HEADERS=true",
+            },
+        ],
+        "logs": [
+            {
+                "evidence_id": "EVID-103",
+                "service_id": "profile-service",
+                "tags": ["benign", "startup"],
+                "text": "profile-service boot completed for synthetic tenant secops-lab",
+            }
+        ],
+        "headers": {
+            "gateway": {
+                "evidence_id": "EVID-104",
+                "present": ["Strict-Transport-Security", "Content-Security-Policy"],
+                "missing": [],
+            }
+        },
+        "dependencies": [
+            {
+                "evidence_id": "EVID-105",
+                "source": "repo_snapshot/requirements.lock",
+                "tags": ["dependency", "benign"],
+                "text": "fastapi==0.115.0; pydantic==2.8.2",
+            }
+        ],
+    },
+    "missing_security_headers_medium": {
+        "task_id": "missing_security_headers_medium",
+        "difficulty": "medium",
+        "title": "Missing security headers",
+        "alert": (
+            "Gateway response-hardening alert: verify whether required security "
+            "headers are missing or weak and submit evidence-backed remediation."
+        ),
+        "ground_truth_id": "GT-HEADERS-001",
+        "finding_type": "missing_security_headers",
+        "severity": "medium",
+        "required_evidence": ["EVID-201"],
+        "remediation_keywords": ["hsts", "csp"],
+        "impact_keywords": ["header", "hsts", "csp", "clickjacking"],
+        "assets": {
+            "services": ["gateway", "profile-service", "admin-service"],
+            "routes": ["/login", "/profile", "/admin/export"],
+            "collections": ["repo_snapshot", "telemetry", "headers", "dependencies"],
+        },
+        "repo": [
+            {
+                "evidence_id": "EVID-202",
+                "source": "repo_snapshot/gateway/security_headers.py",
+                "service_id": "gateway",
+                "tags": ["headers", "config"],
+                "text": "X-Frame-Options is set, but HSTS and CSP are not configured.",
+            }
+        ],
+        "logs": [
+            {
+                "evidence_id": "EVID-203",
+                "service_id": "gateway",
+                "tags": ["benign", "response"],
+                "text": "GET /profile 200 request_id=req-442 synthetic header audit",
+            }
+        ],
+        "headers": {
+            "gateway": {
+                "evidence_id": "EVID-201",
+                "present": ["X-Frame-Options", "X-Content-Type-Options"],
+                "missing": ["Strict-Transport-Security", "Content-Security-Policy"],
+            }
+        },
+        "dependencies": [
+            {
+                "evidence_id": "EVID-204",
+                "source": "repo_snapshot/requirements.lock",
+                "tags": ["dependency", "benign"],
+                "text": "starlette==0.38.2; uvicorn==0.30.1",
+            }
+        ],
+    },
+    "authz_boundary_hard": {
+        "task_id": "authz_boundary_hard",
+        "difficulty": "hard",
+        "title": "Authorisation boundary misconfiguration",
+        "alert": (
+            "Access-control drift alert: investigate a route/role mismatch in the "
+            "synthetic admin service and submit a validated remediation report."
+        ),
+        "ground_truth_id": "GT-AUTHZ-001",
+        "finding_type": "authz_boundary_misconfiguration",
+        "severity": "critical",
+        "required_evidence": ["EVID-301"],
+        "supporting_evidence": ["EVID-302"],
+        "remediation_keywords": ["least privilege", "policy", "regression"],
+        "impact_keywords": ["authorization", "authorisation", "role", "admin"],
+        "assets": {
+            "services": ["gateway", "profile-service", "admin-service"],
+            "routes": ["/login", "/profile", "/admin/export"],
+            "collections": ["repo_snapshot", "telemetry", "headers", "dependencies"],
+        },
+        "repo": [
+            {
+                "evidence_id": "EVID-301",
+                "source": "repo_snapshot/admin-service/policy_matrix.yaml",
+                "service_id": "admin-service",
+                "tags": ["authorization", "role", "policy", "admin export"],
+                "text": (
+                    "route=/admin/export allowed_roles=[admin, analyst] "
+                    "expected_roles=[admin]"
+                ),
+            }
+        ],
+        "logs": [
+            {
+                "evidence_id": "EVID-302",
+                "service_id": "admin-service",
+                "tags": ["authorization", "role", "admin export"],
+                "text": (
+                    "request_id=req-913 route=/admin/export role=analyst "
+                    "decision=allow synthetic boundary-check event"
+                ),
+            },
+            {
+                "evidence_id": "EVID-303",
+                "service_id": "gateway",
+                "tags": ["benign", "auth"],
+                "text": "request_id=req-912 route=/profile role=user decision=allow",
+            },
+        ],
+        "headers": {
+            "admin-service": {
+                "evidence_id": "EVID-304",
+                "present": ["Strict-Transport-Security", "Content-Security-Policy"],
+                "missing": [],
+            }
+        },
+        "dependencies": [
+            {
+                "evidence_id": "EVID-305",
+                "source": "repo_snapshot/requirements.lock",
+                "tags": ["dependency", "benign"],
+                "text": "pyyaml==6.0.2; fastapi==0.115.0",
+            }
+        ],
+    },
+}
+
+
+def list_task_ids() -> list[str]:
+    return list(SCENARIOS)
+
+
+def build_scenario(task_id: str | None, seed: int | None = None) -> dict[str, Any]:
+    """Return a deep-copied scenario with deterministic benign variation."""
+
+    selected_task_id = task_id if task_id in SCENARIOS else DEFAULT_TASK_ID
+    scenario = deepcopy(SCENARIOS[selected_task_id])
+    scenario["seed"] = seed
+
+    rng = Random(seed if seed is not None else 0)
+    service_alias_sets = [
+        ["gateway", "profile-service", "admin-service"],
+        ["edge-gateway", "user-profile", "admin-service"],
+        ["public-gateway", "profile-api", "backoffice-admin"],
+    ]
+    aliases = service_alias_sets[rng.randrange(len(service_alias_sets))]
+    original_services = scenario["assets"]["services"]
+    alias_map = dict(zip(original_services, aliases, strict=True))
+
+    scenario["service_aliases"] = alias_map
+    scenario["assets"]["services"] = [alias_map.get(s, s) for s in original_services]
+
+    for collection_name in ("repo", "logs"):
+        for item in scenario.get(collection_name, []):
+            service_id = item.get("service_id")
+            if service_id in alias_map:
+                item["service_id"] = alias_map[service_id]
+
+    scenario["headers"] = {
+        alias_map.get(service_id, service_id): snapshot
+        for service_id, snapshot in scenario.get("headers", {}).items()
+    }
+
+    for entries_name in ("repo", "logs", "dependencies"):
+        rng.shuffle(scenario.get(entries_name, []))
+
+    return scenario

tests/conftest.py

@@ -0,0 +1,7 @@
+import sys
+from pathlib import Path
+
+
+PACKAGE_PARENT = Path(__file__).resolve().parents[2]
+if str(PACKAGE_PARENT) not in sys.path:
+    sys.path.insert(0, str(PACKAGE_PARENT))

tests/test_environment.py

@@ -0,0 +1,187 @@
+from Cyber_analyst.models import CyberAnalystAction
+from Cyber_analyst.server.Cyber_analyst_environment import CyberAnalystEnvironment
+from Cyber_analyst.server.graders import (
+    grade_authz_boundary_hard,
+    grade_missing_security_headers_medium,
+    grade_secret_exposure_easy,
+    safe_reward,
+)
+
+
+def _run_success_path(task_id, actions):
+    env = CyberAnalystEnvironment()
+    obs = env.reset(task_id=task_id, seed=7)
+    assert obs.task_id == task_id
+
+    for action in actions:
+        obs = env.step(action)
+
+    assert obs.done is True
+    assert obs.tool_result["score"] > 0.5
+    assert 0.01 <= obs.tool_result["score"] <= 0.99
+    assert obs.error == ""
+    return obs
+
+
+def test_secret_exposure_success_path():
+    report = {
+        "findings": [
+            {
+                "finding_type": "secret_exposure",
+                "evidence_ids": ["EVID-101"],
+                "impact": "A synthetic API key secret is exposed in config.",
+                "remediation": "Remove the key and rotate the credential.",
+            }
+        ]
+    }
+    obs = _run_success_path(
+        "secret_exposure_easy",
+        [
+            CyberAnalystAction(tool_name="search_repo", args={"query": "api key"}),
+            CyberAnalystAction(
+                tool_name="create_finding",
+                args={
+                    "finding_type": "secret_exposure",
+                    "evidence_ids": ["EVID-101"],
+                    "severity_guess": "high",
+                    "remediation": "Remove and rotate the synthetic credential.",
+                },
+            ),
+            CyberAnalystAction(tool_name="validate_finding", args={"finding_id": "FND-001"}),
+            CyberAnalystAction(tool_name="submit_report", args={"report_json": report}),
+        ],
+    )
+    assert obs.verified_findings[0]["matching_gt_id"] == "GT-SECRET-001"
+    assert "trajectory_jsonl" in obs.tool_result
+    assert "search_repo" in obs.tool_result["trajectory_jsonl"]
+
+
+def test_missing_security_headers_success_path():
+    report = {
+        "findings": [
+            {
+                "finding_type": "missing_security_headers",
+                "evidence_ids": ["EVID-201"],
+                "impact": "The gateway is missing HSTS and CSP headers.",
+                "remediation": "Add HSTS and CSP at the gateway.",
+            }
+        ]
+    }
+    obs = _run_success_path(
+        "missing_security_headers_medium",
+        [
+            CyberAnalystAction(
+                tool_name="check_security_headers", args={"service_id": "gateway"}
+            ),
+            CyberAnalystAction(
+                tool_name="create_finding",
+                args={
+                    "finding_type": "missing_security_headers",
+                    "evidence_ids": ["EVID-201"],
+                    "severity_guess": "medium",
+                    "remediation": "Add HSTS and CSP headers.",
+                },
+            ),
+            CyberAnalystAction(tool_name="validate_finding", args={"finding_id": "FND-001"}),
+            CyberAnalystAction(tool_name="submit_report", args={"report_json": report}),
+        ],
+    )
+    assert obs.score_breakdown["valid_evidence"] == 0.15
+
+
+def test_authz_boundary_success_path_with_alias_compatible_service_ids():
+    report = {
+        "findings": [
+            {
+                "finding_type": "authz_boundary_misconfiguration",
+                "evidence_ids": ["EVID-301", "EVID-302"],
+                "impact": "The admin route authorization policy allows an analyst role.",
+                "remediation": "Apply least privilege in the policy and add a regression test.",
+            }
+        ]
+    }
+    obs = _run_success_path(
+        "authz_boundary_hard",
+        [
+            CyberAnalystAction(tool_name="list_assets", args={}),
+            CyberAnalystAction(
+                tool_name="get_log_events",
+                args={"service_id": "admin-service", "query": "admin export"},
+            ),
+            CyberAnalystAction(tool_name="search_repo", args={"query": "admin export"}),
+            CyberAnalystAction(
+                tool_name="create_finding",
+                args={
+                    "finding_type": "authz_boundary_misconfiguration",
+                    "evidence_ids": ["EVID-301", "EVID-302"],
+                    "severity_guess": "critical",
+                    "remediation": "Apply least privilege and add a regression test.",
+                },
+            ),
+            CyberAnalystAction(tool_name="validate_finding", args={"finding_id": "FND-001"}),
+            CyberAnalystAction(tool_name="submit_report", args={"report_json": report}),
+        ],
+    )
+    assert obs.score_breakdown["actionable_remediation"] == 0.15
+
+
+def test_invalid_tool_returns_observation_error():
+    env = CyberAnalystEnvironment()
+    env.reset(task_id="secret_exposure_easy", seed=1)
+    obs = env.step(CyberAnalystAction(tool_name="shell", args={"cmd": "whoami"}))
+    assert obs.done is False
+    assert obs.error == "unsupported_tool"
+    assert obs.tool_result["ok"] is False
+
+
+def test_hallucinated_report_scores_low_but_in_range():
+    env = CyberAnalystEnvironment()
+    env.reset(task_id="secret_exposure_easy", seed=1)
+    obs = env.step(
+        CyberAnalystAction(
+            tool_name="submit_report",
+            args={
+                "report_json": {
+                    "findings": [
+                        {
+                            "finding_type": "remote_code_execution",
+                            "evidence_ids": [],
+                            "impact": "Unsupported claim.",
+                            "remediation": "Unsupported remediation.",
+                        }
+                    ]
+                }
+            },
+        )
+    )
+    assert obs.done is True
+    assert obs.tool_result["score"] == 0.01
+
+
+def test_repeated_action_hard_stops_episode():
+    env = CyberAnalystEnvironment()
+    env.reset(task_id="secret_exposure_easy", seed=1)
+    obs = None
+    for _ in range(6):
+        obs = env.step(CyberAnalystAction(tool_name="list_assets", args={}))
+    assert obs is not None
+    assert obs.done is True
+    assert obs.error == "repeat_hard_stop"
+
+
+def test_seed_determinism_for_assets():
+    env_one = CyberAnalystEnvironment()
+    env_two = CyberAnalystEnvironment()
+    env_one.reset(task_id="authz_boundary_hard", seed=22)
+    env_two.reset(task_id="authz_boundary_hard", seed=22)
+    obs_one = env_one.step(CyberAnalystAction(tool_name="list_assets", args={}))
+    obs_two = env_two.step(CyberAnalystAction(tool_name="list_assets", args={}))
+    assert obs_one.tool_result == obs_two.tool_result
+
+
+def test_grader_adapters_and_clamp_are_strictly_in_range():
+    assert safe_reward(-1) == 0.01
+    assert safe_reward(2) == 0.99
+    assert 0.01 <= grade_secret_exposure_easy() <= 0.99
+    assert 0.01 <= grade_missing_security_headers_medium() <= 0.99
+    assert 0.01 <= grade_authz_boundary_hard() <= 0.99

tests/test_inference.py

@@ -0,0 +1,47 @@
+import pytest
+
+from Cyber_analyst.inference import (
+    ModelActionError,
+    action_to_log,
+    error_action,
+    parse_model_action,
+)
+
+
+def test_parse_model_action_accepts_compact_json():
+    action = parse_model_action('{"tool_name":"search_repo","args":{"query":"api key"}}')
+
+    assert action.tool_name == "search_repo"
+    assert action.args == {"query": "api key"}
+
+
+def test_parse_model_action_accepts_fenced_json():
+    action = parse_model_action(
+        """```json
+{"tool_name":"list_assets","args":{}}
+```"""
+    )
+
+    assert action.tool_name == "list_assets"
+    assert action.args == {}
+
+
+def test_parse_model_action_rejects_malformed_json():
+    with pytest.raises(ModelActionError, match="model_parse_error"):
+        parse_model_action("search the repo for api keys")
+
+
+def test_action_to_log_is_single_line_json():
+    action = parse_model_action('{"tool_name":"search_repo","args":{"query":"api\\nkey"}}')
+
+    logged = action_to_log(action)
+
+    assert "\n" not in logged
+    assert logged == '{"args":{"query":"api\\nkey"},"tool_name":"search_repo"}'
+
+
+def test_error_action_uses_strict_diagnostic_tool_name():
+    action = error_action(ModelActionError("model_parse_error: empty response"))
+
+    assert action.tool_name == "model_parse_error"
+    assert action.args == {"message": "model_parse_error: empty response"}