Upload folder using huggingface_hub

README.md

@@ -1,158 +1,227 @@
----
-title: Cyber Analyst Environment Server
-emoji: 🎯
-colorFrom: pink
-colorTo: red
-sdk: docker
-pinned: false
-app_port: 8000
-base_path: /web
-tags:
-  - openenv
----
-
-# Cyber Analyst Environment
-
-Cyber Analyst is an OpenEnv implementation of the "SecOps Evidence Gym" design from `docs/deep-research-report.md`. It benchmarks a bounded, safe security triage workflow: investigate synthetic artifacts, cite evidence IDs, validate candidate findings with deterministic verifiers, and submit a remediation report.
-
-The environment contains no live targets, no real secrets, no exploit workflow, no shell, and no outbound investigation tools. All evidence is static synthetic lab data.
-
-## Tasks
-
-The manifest ships three graded tasks:
-
-| Task id | Difficulty | Goal |
-| --- | --- | --- |
-| `secret_exposure_easy` | easy | Find a synthetic API-key-like secret in a repo snapshot and propose removal plus rotation. |
-| `missing_security_headers_medium` | medium | Detect missing HSTS/CSP headers in a synthetic gateway header snapshot. |
-| `authz_boundary_hard` | hard | Detect an admin route role-policy mismatch without exploitation. |
-
-## Action Contract
-
-Use one bounded tool call per `step`:
-
-```python
-CyberAnalystAction(
-    tool_name="search_repo",
-    args={"query": "api key"},
-)
-```
-
-Approved tools:
-
-- `list_assets()`
-- `get_log_events(service_id, query)`
-- `check_security_headers(service_id)`
-- `search_repo(query)`
-- `scan_dependencies()`
-- `create_finding(finding_type, evidence_ids, severity_guess, remediation)`
-- `validate_finding(finding_id)`
-- `submit_report(report_json)`
-
-## Observation Contract
-
-Each observation includes:
-
-- `alert`: task prompt
-- `tool_catalog`: approved tool list
-- `tool_result`: latest tool result
-- `evidence_ids`: discovered evidence IDs
-- `candidate_findings`: created findings
-- `verified_findings`: verifier-confirmed findings
-- `score_breakdown`: deterministic scoring explanation
-- `step_budget_remaining`, `error`, `done`, and `reward`
-
-Rewards and final scores are clamped to `0.01..0.99` for validator compatibility.
-
-`submit_report` also returns `trajectory_jsonl`, a JSONL export of the episode
-events up to report submission. This is intended for offline inspection and
-future training data extraction.
-
-## Local Run
-
-From this directory:
-
-```bash
-uv run server
-```
-
-Then connect with the client:
-
-```python
-from Cyber_analyst import CyberAnalystAction, CyberAnalystEnv
-
-with CyberAnalystEnv(base_url="http://localhost:8000").sync() as env:
-    result = env.reset(task_id="secret_exposure_easy", seed=7)
-    result = env.step(CyberAnalystAction(tool_name="search_repo", args={"query": "api key"}))
-    print(result.observation.tool_result)
-```
-
-## Baseline Inference
-
-`inference.py` runs a deterministic scripted baseline and prints strict parser-friendly logs:
-
-```text
-[START] task=<task_id> env=Cyber_analyst model=<model_name>
-[STEP] step=<n> action=<tool_name> reward=<0.00> done=<true|false> error=<msg|null>
-[END] task=<task_id> success=<true|false> steps=<n> score=<0.00> rewards=<r1,r2,...>
-```
-
-LLM calls are not enabled by default. The script already includes OpenAI SDK configuration compatible with Hugging Face Inference Providers so model-backed report drafting can be added later:
-
-```bash
-set ENV_URL=http://localhost:8000
-set API_BASE_URL=https://router.huggingface.co/v1
-set MODEL_NAME=openai/gpt-oss-120b:novita
-set HF_TOKEN=<your-hugging-face-token>
-python inference.py
-```
-
-## Validation
-
-Useful local checks:
-
-```bash
-python -m py_compile server/Cyber_analyst_environment.py inference.py
-python -m pytest tests
-.\.venv\Scripts\openenv.exe validate . --json
-```
-
-## Docker
-
-Build the environment image from this directory:
-
-```bash
-docker build -t cyber-analyst-env:latest -f server/Dockerfile .
-```
-
-Run:
-
-```bash
-docker run -p 8000:8000 cyber-analyst-env:latest
-```
-
-Health check:
-
-```bash
-curl http://localhost:8000/health
-```
-
-## Deployment
-
-Deploy to Hugging Face Spaces with OpenEnv:
-
-```bash
-openenv push --repo-id <your-hf-username>/Cyber_analyst
-```
-
-The deployed Space exposes `/health`, `/docs`, `/ws`, and the optional `/web` interface when web UI support is enabled by the OpenEnv runtime.
-
-## Adding Scenarios
-
-Add new safe scenarios in `server/tasks.py` by extending `SCENARIOS` with:
-
-- a stable `task_id`
-- synthetic `assets`, `repo`, `logs`, `headers`, and `dependencies` entries
-- `ground_truth_id`, `finding_type`, `required_evidence`, `impact_keywords`, and `remediation_keywords`
-
-Then add a grader adapter in `server/graders.py` and a matching `tasks` entry in `openenv.yaml`. Keep all artifacts synthetic, keep correctness deterministic, and avoid adding real targets or arbitrary execution tools.
+---
+title: Cyber Analyst Environment Server
+emoji: 🎯
+colorFrom: pink
+colorTo: red
+sdk: docker
+pinned: false
+app_port: 8000
+base_path: /web
+tags:
+  - openenv
+---
+
+# Cyber Analyst Environment
+
+Cyber Analyst is an OpenEnv implementation of the "SecOps Evidence Gym". It benchmarks a bounded, safe security-triage workflow: investigate synthetic artifacts, cite evidence IDs, validate candidate findings with deterministic verifiers, and submit a remediation report.
+
+The environment contains no live targets, no real secrets, no exploit workflow, no shell, and no outbound investigation tools. All evidence is static synthetic lab data.
+
+## Motivation
+
+Frontier models are becoming much stronger at security-relevant reasoning. Anthropic's April 7, 2026 report, [Assessing Claude Mythos Preview's cybersecurity capabilities](https://red.anthropic.com/2026/mythos-preview/), describes a model that can identify and exploit subtle vulnerabilities across real software targets, and argues that the same capability jump should be directed toward defense.
+
+That creates a practical gap: many modern applications are built quickly, including "vibe coded" apps whose security review may not keep pace with generation speed. This environment is a small, safe training and evaluation surface for the defensive side of that gap. The goal is to help train and benchmark smaller, more accessible models to behave like careful application-security analysts: gather evidence, avoid unsupported claims, validate findings, and recommend concrete fixes.
+
+## Environment Description
+
+Each episode simulates a synthetic microservice organization with three services:
+
+- `gateway`
+- `profile-service`
+- `admin-service`
+
+The agent starts from an alert and can inspect only closed-world artifact collections:
+
+- `repo_snapshot`: static code/config snippets
+- `telemetry`: sanitized log events
+- `headers`: static response-header snapshots
+- `dependencies`: static dependency manifest excerpts
+
+The episode budget is 12 steps. Seeds deterministically vary benign details such as service aliases and evidence ordering while keeping the same task ground truth reproducible.
+
+## Tasks
+
+The manifest ships three graded tasks:
+
+| Task id | Difficulty | Task description | Expected difficulty |
+| --- | --- | --- | --- |
+| `secret_exposure_easy` | easy | Find a synthetic API-key-like secret in a repo snapshot and propose removal plus rotation. | Easiest path: one focused `search_repo` call can surface the relevant evidence, then the agent must create, validate, and report the finding. |
+| `missing_security_headers_medium` | medium | Detect missing HSTS/CSP headers in a synthetic gateway header snapshot. | Requires choosing the purpose-built `check_security_headers` tool and mapping missing headers to remediation instead of over-searching unrelated artifacts. |
+| `authz_boundary_hard` | hard | Detect an admin route role-policy mismatch without exploitation. | Requires correlating route/role policy evidence with a supporting log event and recommending least-privilege policy remediation plus regression testing. |
+
+## Action Space
+
+Each `step` accepts exactly one bounded simulator tool call:
+
+```python
+CyberAnalystAction(
+    tool_name="search_repo",
+    args={"query": "api key"},
+)
+```
+
+Approved tools:
+
+| Tool | Arguments | Purpose |
+| --- | --- | --- |
+| `list_assets` | `{}` | List synthetic services, routes, and artifact collections. |
+| `get_log_events` | `{"service_id": "str", "query": "str"}` | Return sanitized telemetry evidence IDs for a service/query. |
+| `check_security_headers` | `{"service_id": "str"}` | Inspect a service header snapshot and return pass/fail evidence. |
+| `search_repo` | `{"query": "str"}` | Search synthetic repo/config snippets for evidence IDs. |
+| `scan_dependencies` | `{}` | Inspect a synthetic dependency manifest excerpt. |
+| `create_finding` | `{"finding_type": "str", "evidence_ids": ["str"], "severity_guess": "str", "remediation": "str"}` | Store a candidate finding for verifier review. |
+| `validate_finding` | `{"finding_id": "str"}` | Run the deterministic verifier for a candidate finding. |
+| `submit_report` | `{"report_json": {"findings": [...]}}` | Submit the final structured report and end the episode. |
+
+Unsupported tools return an observation error instead of running arbitrary commands. Repeating the exact same action is penalized, and six repeated identical actions hard-stop the episode.
+
+## Observation Space
+
+Each observation is a `CyberAnalystObservation` with:
+
+| Field | Definition |
+| --- | --- |
+| `task_id` | Current benchmark task ID. |
+| `alert` | Initial alert or task prompt. |
+| `phase` | Current episode phase, usually `investigate` or `done`. |
+| `tool_catalog` | Approved tool list and argument schemas. |
+| `tool_result` | Result returned by the latest tool call. |
+| `evidence_ids` | Evidence IDs discovered so far. |
+| `candidate_findings` | Candidate findings created by the agent. |
+| `verified_findings` | Verifier-confirmed findings. |
+| `step_budget_remaining` | Steps remaining before timeout. |
+| `score_breakdown` | Deterministic final scoring explanation after report submission. |
+| `error` | Non-fatal environment error, if any. |
+| `done` | Whether the episode has ended. |
+| `reward` | Step reward clamped to the validator-compatible range. |
+
+`submit_report` also returns `trajectory_jsonl`, a JSONL export of the episode events up to report submission. This is intended for offline inspection and future training data extraction.
+
+## Scoring
+
+Final reports are scored deterministically:
+
+- base score: `0.05`
+- verified correct finding with matching report impact: `+0.60`
+- valid evidence ID in the report: `+0.15`
+- actionable remediation keywords: `+0.15`
+- hallucinated or unverified finding claims: `-0.40` each
+- submitting without verifier validation: `-0.20`
+
+Rewards and final scores are clamped to `0.01..0.99` for validator compatibility.
+
+## Baseline Scores
+
+The current deterministic oracle rollout follows the intended evidence -> finding -> validation -> report path for each task. These scores were measured locally against the environment with `seed=7`.
+
+| Task id | Baseline type | Steps | Final score | Step rewards |
+| --- | --- | ---: | ---: | --- |
+| `secret_exposure_easy` | deterministic oracle | 4 | `0.95` | `0.05, 0.06, 0.11, 0.98` |
+| `missing_security_headers_medium` | deterministic oracle | 4 | `0.95` | `0.05, 0.06, 0.11, 0.98` |
+| `authz_boundary_hard` | deterministic oracle | 6 | `0.95` | `0.03, 0.05, 0.05, 0.06, 0.11, 0.98` |
+
+A hallucinated one-step report scores `0.01`; repeated identical actions hard-stop at a low score.
+
+## Setup
+
+From this directory, install dependencies:
+
+```bash
+uv sync
+```
+
+Run the local server:
+
+```bash
+uv run server
+```
+
+Health check:
+
+```bash
+curl http://localhost:8000/health
+```
+
+Then connect with the client:
+
+```python
+from Cyber_analyst import CyberAnalystAction, CyberAnalystEnv
+
+with CyberAnalystEnv(base_url="http://localhost:8000").sync() as env:
+    result = env.reset(task_id="secret_exposure_easy", seed=7)
+    result = env.step(CyberAnalystAction(tool_name="search_repo", args={"query": "api key"}))
+    print(result.observation.tool_result)
+```
+
+## Baseline Inference
+
+`inference.py` runs a model-backed baseline over the configured task set and prints strict parser-friendly logs:
+
+```text
+[START] task=<task_id> env=Cyber_analyst model=<model_name>
+[STEP] step=<n> action=<compact_json_action> reward=<0.00> done=<true|false> error=<msg|null>
+[END] task=<task_id> success=<true|false> steps=<n> score=<0.00> rewards=<r1,r2,...>
+```
+
+The script uses the OpenAI SDK with Hugging Face Inference Providers by default:
+
+```powershell
+$env:ENV_URL = "http://localhost:8000"
+$env:API_BASE_URL = "https://router.huggingface.co/v1"
+$env:MODEL_NAME = "google/gemma-4-31B-it:fastest"
+$env:HF_TOKEN = "<your-hugging-face-token>"
+python inference.py
+```
+
+Use `$env:TASK_NAME = "<task_id>"` to run one task instead of all three.
+
+## Validation
+
+Useful local checks:
+
+```bash
+python -m py_compile server/Cyber_analyst_environment.py inference.py
+python -m pytest tests
+.\.venv\Scripts\openenv.exe validate . --json
+```
+
+## Docker
+
+Build the environment image from this directory:
+
+```bash
+docker build -t cyber-analyst-env:latest -f server/Dockerfile .
+```
+
+Run:
+
+```bash
+docker run -p 8000:8000 cyber-analyst-env:latest
+```
+
+Health check:
+
+```bash
+curl http://localhost:8000/health
+```
+
+## Deployment
+
+Deploy to Hugging Face Spaces with OpenEnv:
+
+```bash
+openenv push --repo-id <your-hf-username>/Cyber_analyst
+```
+
+The deployed Space exposes `/health`, `/docs`, `/ws`, and the optional `/web` interface when web UI support is enabled by the OpenEnv runtime.
+
+## Adding Scenarios
+
+Add new safe scenarios in `server/tasks.py` by extending `SCENARIOS` with:
+
+- a stable `task_id`
+- synthetic `assets`, `repo`, `logs`, `headers`, and `dependencies` entries
+- `ground_truth_id`, `finding_type`, `required_evidence`, `impact_keywords`, and `remediation_keywords`
+
+Then add a grader adapter in `server/graders.py` and a matching `tasks` entry in `openenv.yaml`. Keep all artifacts synthetic, keep correctness deterministic, and avoid adding real targets or arbitrary execution tools.

__init__.py

@@ -1,17 +1,17 @@
-# Copyright (c) Meta Platforms, Inc. and affiliates.
-# All rights reserved.
-#
-# This source code is licensed under the BSD-style license found in the
-# LICENSE file in the root directory of this source tree.
-
-"""Cyber Analyst Environment."""
-
-from .client import CyberAnalystEnv
-from .models import CyberAnalystAction, CyberAnalystObservation, CyberAnalystState
-
-__all__ = [
-    "CyberAnalystAction",
-    "CyberAnalystObservation",
-    "CyberAnalystState",
-    "CyberAnalystEnv",
-]
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+"""Cyber Analyst Environment."""
+
+from .client import CyberAnalystEnv
+from .models import CyberAnalystAction, CyberAnalystObservation, CyberAnalystState
+
+__all__ = [
+    "CyberAnalystAction",
+    "CyberAnalystObservation",
+    "CyberAnalystState",
+    "CyberAnalystEnv",
+]

client.py

@@ -1,39 +1,39 @@
-# Copyright (c) Meta Platforms, Inc. and affiliates.
-# All rights reserved.
-#
-# This source code is licensed under the BSD-style license found in the
-# LICENSE file in the root directory of this source tree.
-
-"""Cyber Analyst Environment client."""
-
-from typing import Any
-
-from openenv.core import EnvClient
-from openenv.core.client_types import StepResult
-
-from .models import CyberAnalystAction, CyberAnalystObservation, CyberAnalystState
-
-
-class CyberAnalystEnv(
-    EnvClient[CyberAnalystAction, CyberAnalystObservation, CyberAnalystState]
-):
-    """WebSocket client for the Cyber Analyst OpenEnv environment."""
-
-    def _step_payload(self, action: CyberAnalystAction) -> dict[str, Any]:
-        return action.model_dump(exclude_none=True)
-
-    def _parse_result(
-        self, payload: dict[str, Any]
-    ) -> StepResult[CyberAnalystObservation]:
-        obs_data = dict(payload.get("observation", {}))
-        obs_data["done"] = payload.get("done", False)
-        obs_data["reward"] = payload.get("reward")
-        observation = CyberAnalystObservation.model_validate(obs_data)
-        return StepResult(
-            observation=observation,
-            reward=payload.get("reward"),
-            done=payload.get("done", False),
-        )
-
-    def _parse_state(self, payload: dict[str, Any]) -> CyberAnalystState:
-        return CyberAnalystState.model_validate(payload)
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+"""Cyber Analyst Environment client."""
+
+from typing import Any
+
+from openenv.core import EnvClient
+from openenv.core.client_types import StepResult
+
+from .models import CyberAnalystAction, CyberAnalystObservation, CyberAnalystState
+
+
+class CyberAnalystEnv(
+    EnvClient[CyberAnalystAction, CyberAnalystObservation, CyberAnalystState]
+):
+    """WebSocket client for the Cyber Analyst OpenEnv environment."""
+
+    def _step_payload(self, action: CyberAnalystAction) -> dict[str, Any]:
+        return action.model_dump(exclude_none=True)
+
+    def _parse_result(
+        self, payload: dict[str, Any]
+    ) -> StepResult[CyberAnalystObservation]:
+        obs_data = dict(payload.get("observation", {}))
+        obs_data["done"] = payload.get("done", False)
+        obs_data["reward"] = payload.get("reward")
+        observation = CyberAnalystObservation.model_validate(obs_data)
+        return StepResult(
+            observation=observation,
+            reward=payload.get("reward"),
+            done=payload.get("done", False),
+        )
+
+    def _parse_state(self, payload: dict[str, Any]) -> CyberAnalystState:
+        return CyberAnalystState.model_validate(payload)

docs/deep-research-report.md

@@ -1,531 +1,531 @@
-# OpenEnv Hackathon Execution Pipeline for a Safe Cybersecurity Analyst Environment
-
-## Executive decision and scope selection
-
-**SECTION 1 — Executive Decision**
-
-[SOURCED] The hackathon’s *validator + judging* constraints strongly favour environments that: (a) simulate a real-world task (not “games/toys”), (b) are fully OpenEnv-compliant, (c) ship with **≥3 tasks with graders**, (d) produce **scores/rewards in the 0–1 range**, (e) include a reproducible `inference.py` that uses the **OpenAI client** (for any LLM calls) and prints strict `[START]/[STEP]/[END]` logs, and (f) run within a ~**20 minute** budget on ~**2 vCPU / 8 GB** infra. citeturn3view6turn3view7turn22view1turn22view2  
-
-[INFERENCE] Under these realities, your cybersecurity-analyst direction is *not* automatically the best, but it *can* become a high-probability-to-win choice if—and only if—you narrow to a deterministic, bounded, “investigate → cite evidence → verify → remediate” loop where (i) tools are tightly sandboxed, (ii) graders are deterministic, and (iii) the action space is small enough to be learnable and demo-able under the runtime cap.
-
-[PROPOSAL] **Decision:** keep the cybersecurity direction, but **narrow aggressively** to a V1 environment that benchmarks **disciplined security triage + evidence-grounded reporting**, not pentesting/exploitation. The V1 I recommend building is:
-
-[PROPOSAL] **“SecOps Evidence Gym”** — a safe, isolated OpenEnv environment where an agent investigates a *synthetic* microservice “organisation” via a **bounded tool API**, collects **evidence IDs**, validates candidate findings through **deterministic verifiers**, and submits a structured remediation report.  
-
-[SOURCED] This matches strong “winner DNA” seen in `kube-sre-gym` (realistic professional workflow + verification + narrative clarity) while remaining implementable in a hackathon budget. citeturn10view0turn18view0  
-
-[PROPOSAL] **What to cut entirely in V1 (non-negotiable):**  
-- “Live target” behaviour; no external network targets; no arbitrary HTTP to the internet. 🔒  
-- Any exploit payload recipes, exploit chains, privilege-escalation playbooks, or “how to hack X”. 🔒  
-- Arbitrary shell access (`bash`, `kubectl`, `nmap`, etc.) inside the environment. (Action space explosion + safety risk.)  
-- LLM-only grading/judging for correctness. (Reward hacking + non-determinism.)  
-
-[SOURCED] **What to keep (but narrow):** tool-using investigation, multi-step interaction, and deterministic verification—these are consistent with what OpenEnv is designed to support (typed `reset/step/state`, isolated server, type-safe schemas). citeturn18view0turn19search1  
-
-**SECTION 3 — Candidate Scope Comparison**
-
-[SOURCED] The scoring below is anchored on hackathon validator requirements (3+ graded tasks, 0–1 scoring, strict inference logging, runtime limits) plus OpenEnv’s scaffolding/CLI/deployment model. citeturn3view6turn18view0turn22view1  
-
-[PROPOSAL] Weighted criteria (sum=1.00): judging fit 0.14, OpenEnv fit 0.12, grader determinism 0.14, implementation risk 0.12, runtime feasibility 0.10, demoability 0.10, real-world usefulness 0.10, novelty 0.08, training usefulness 0.06, shipping-on-time likelihood 0.04.
-
-| Candidate scope | Judging fit | OpenEnv fit | Determinism | Impl risk (lower=better) | Runtime | Demo | Real-world use | Novelty | Training use | Ship-on-time | Weighted total |
-|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
-| **A. Your original direction:** “disciplined cyber analyst investigating a sandbox” (broad) | 8 | 7 | 6 | 4 | 6 | 8 | 8 | 8 | 8 | 4 | **6.7** |
-| **B. Narrow cyber variant (recommended):** evidence-first triage lab with bounded tools + deterministic verifiers + structured report | 9 | 8 | 9 | 7 | 9 | 9 | 8 | 7 | 8 | 8 | **8.4** |
-| **C. Adjacent: SRE incident triage (single-turn, deterministic logs → RCA)** | 9 | 8 | 10 | 9 | 10 | 8 | 8 | 5 | 6 | 9 | **8.3** |
-| **D. Adjacent: prompt-injection “WAF” benchmark** | 7 | 8 | 8 | 7 | 9 | 9 | 7 | 7 | 6 | 7 | **7.6** |
-
-[INFERENCE] Candidate C (SRE triage) is extremely validator-safe (many examples already pass deep validation), but it is likely more saturated and less “new” in judging. Candidate B keeps your cybersecurity theme while retaining the determinism and boundedness that the validator and time budget demand, so it is the best balance for **winning + real-world usefulness**.
-
-**SECTION 4 — Final V1 Problem Statement**
-
-[PROPOSAL] **One-sentence version:** Build a safe OpenEnv environment that trains and benchmarks an agent to perform **evidence-grounded security triage** on a bounded synthetic system and produce a **remediation-oriented report** without hallucinating.
-
-[PROPOSAL] **Short pitch (demo-ready):** “SecOps Evidence Gym” gives the model an alert and a constrained set of investigation tools. The model must gather evidence, validate findings with deterministic verifiers, and submit a structured report. Scores reward verified correctness, penalise hallucinated claims and wasted steps, and remain strictly within (0,1) to satisfy the hackathon validator. ✅🔒
-
-[PROPOSAL] **Precise implementation version:** A FastAPI OpenEnv server exposing typed `reset()`, `step()`, `state()` and a manifest `openenv.yaml` with at least three tasks (easy/medium/hard), each associated with a grader. The environment implements multi-step tool calls inside `step()`, and uses deterministic verifiers plus a strict score clamp to `(0,1)` for validator compatibility. citeturn18view0turn19search1turn23view0turn26view0turn27search0  
-
-## Winner patterns and judging fit extraction
-
-**SECTION 2 — Winner Pattern Extraction**
-
-[SOURCED] `kube-sre-gym` (OpenEnv Hackathon SF winner) demonstrates several patterns that appear strongly aligned with what judges value: a **realistic professional task**, an explicit **multi-step workflow** (triage → investigate → fix → verify), **multi-layer verification** (programmatic health checks + judge), a strong narrative that explains learning dynamics, and deployment on **Hugging Face Spaces**. citeturn10view0turn14view0  
-
-image_group{"layout":"carousel","aspect_ratio":"16:9","query":["kube-sre-gym OpenEnv Hackathon winner screenshot","OpenEnv framework architecture diagram","Hugging Face Spaces OpenEnv environment web demo","Incident Triage Environment OpenEnv Hackathon 2026 screenshot"],"num_per_query":1}
-
-[SOURCED] Concretely, `kube-sre-gym` highlights: “real cluster/tool interaction”, verification layers to prevent false success, curriculum progression, and strong documentation that makes the environment’s value obvious quickly. citeturn10view0  
-
-[SOURCED] A 2026 hackathon submission that explicitly claims Phase-2 deep-validation success (`Incident-Triage-Environment`) reveals particularly transferable “validator-winning” patterns:  
-- A manifest with `spec_version`, `runtime`, `app`, `port`, and a **`tasks:` list where each task has a `grader:` pointing to importable Python functions**. citeturn23view0  
-- Deterministic graders that clamp outputs to a validator-friendly range. citeturn26view0turn26view3  
-- An `inference.py` that uses the OpenAI SDK and prints the strict stdout protocol with `[START]`, `[STEP]`, `[END]` lines in a stable key=value format. citeturn22view1turn22view2turn22view4  
-
-[SOURCED] The official OpenEnv repo reinforces what is transferable: type-safe action/observation/state contracts, containerised isolation, and standard Gym-like APIs. It also explicitly says OpenEnv is experimental and APIs can change, which increases the value of a minimal, validator-first build loop. citeturn18view0turn19search2  
-
-[INFERENCE] Given hackathon evaluation combines programmatic checks with LLM scoring, you must optimise for **deterministic correctness** *and* a compelling narrative/demo. citeturn3view7turn3view6  
-
-[PROPOSAL] Transferable “winner patterns” you should copy (selectively):  
-- **Strong “professional workflow” framing** (SRE, security analyst, triage) with clear step boundaries.  
-- **Small, discoverable tool set** that mirrors real practice (logs, config, policy checks) while staying bounded.  
-- **Deterministic verification** (programmatic checks) as the source of truth for correctness.  
-- **Narrative traceability**: logs, episode IDs, and a short “watch the agent work” demo.  
-- **Deployment excellence**: clean Docker build, working `/health`, working `/web` UI if enabled, and reproducible inference.
-
-[PROPOSAL] What *not* to copy blindly:  
-- The “real cluster” dependency (e.g., GKE) is high operational burden and can fail the hackathon’s limited infra budget. citeturn10view0turn3view6  
-- LLM-as-judge for correctness (too easy to reward-hack; non-deterministic). (Use it, at most, for *format/style*, not correctness.)  
-
-## Core V1 environment design and benchmark tasks
-
-**SECTION 8 — Core Environment Design**
-
-**V1 concept (aggressively narrow).**  
-[PROPOSAL] Your environment is a **synthetic organisation** with a small, fixed topology (three “services” + artefacts). The agent receives an alert. It can only interact via **approved tools** (implemented inside the simulator). It must (a) gather evidence IDs, (b) validate candidate findings, and (c) submit a report.
-
-**Topology (V1).**  
-[PROPOSAL] Fixed components (no containers inside containers):  
-- `gateway` (public entry), `profile-service`, `admin-service`  
-- `repo_snapshot` (static code/config excerpts)  
-- `telemetry` (sanitised logs + “header snapshot” + “dependency manifest snapshot”)  
-
-**Reset logic.**  
-[PROPOSAL] `reset(task_id=..., seed=...)` selects a scenario variant and initialises:  
-- episode ID, step count  
-- scenario ground truth (one injected issue per episode in V1)  
-- tool budgets + “allowed scope” banner  
-- an evidence registry mapping `EVID-### → artefact snippet`  
-Return an initial observation containing the alert, the tool catalogue, and an empty “verified findings” list.
-
-**Randomisation strategy.**  
-[PROPOSAL] Use seed-driven, deterministic randomisation:  
-- rename services/routes/IDs (`profile-service` might become `user-profile`),  
-- shuffle benign log lines around the key evidence,  
-- vary exact header sets / dependency versions within a small closed set,  
-- keep each scenario **fully reproducible from the seed**.
-
-[SOURCED] Benchmark generators (e.g., AMaze) exist specifically to create diverse but controlled environments for evaluating generalisation, supporting the idea of seeded procedural variation rather than a single static scenario. citeturn16search7turn16search1  
-
-**Safety boundaries.**  
-[PROPOSAL] The sandbox contains **no live targets**, no real secrets, and no outbound network. “Secrets” are synthetic strings with an explicit “DO NOT USE OUTSIDE LAB” marker. Tools return synthetic results only. 🔒  
-
-[SOURCED] NIST’s cyber range guidance emphasises cyber ranges as safe and legal environments for training and assessment; separate research also discusses that cyber ranges themselves have security risks that must be mitigated (e.g., leakage/misuse), reinforcing the need for strict isolation and artefact sanitisation. citeturn29search1turn29search2  
-
-**How state is exposed to the agent.**  
-[PROPOSAL] Expose only a concise state summary: current phase, step budget remaining, tools remaining, verified findings count, and recent evidence IDs. Keep full ground truth hidden.
-
-**Tool/action design (bounded action space).**  
-[PROPOSAL] V1 tool list (keep it ≤8 tools):  
-1) `list_assets()` → returns asset IDs and route IDs  
-2) `get_log_events(service_id, query)` → returns evidence IDs  
-3) `check_security_headers(service_id)` → returns evidence IDs + pass/fail list  
-4) `search_repo(query)` → returns evidence IDs from code snippets  
-5) `scan_dependencies()` → returns evidence IDs from a lockfile excerpt  
-6) `create_finding(finding_type, evidence_ids, severity_guess, remediation)` → stores candidate finding  
-7) `validate_finding(finding_id)` → deterministic verifier; returns `(verified, matching_gt_id)`  
-8) `submit_report(report_json)` → terminal action  
-
-**Anti-loop logic.**  
-[PROPOSAL] Track action signatures `(tool_name, args_hash)` and:  
-- apply increasing penalties for repeats,  
-- hard-stop an episode if identical actions repeat ≥6 times, returning `done=True` with a low score,  
-- always return a valid observation (never a server crash) to preserve training rollouts.
-
-[SOURCED] OpenEnv’s environment-creation guidance strongly implies you should implement robust behaviour around `reset/step/state` with typed contracts and predictable server behaviour. citeturn19search1turn18view0  
-
-**SECTION 9 — Tasks / Benchmarks**
-
-[SOURCED] The hackathon requires **at least 3 tasks with graders** and explicitly checks the tasks registry. citeturn3view6turn27search0  
-
-[PROPOSAL] V1 ships exactly **3 flagship tasks**, difficulty-tiered, each with deterministic success criteria and intermediate milestones.
-
-**Flagship tasks (easy/medium/hard).**  
-[PROPOSAL] Each task is a *family* with small seeded variants.
-
-**Easy: Secret exposure in repo snapshot**  
-- Goal: identify a leaked synthetic API key in a config file excerpt; propose rotation/removal.  
-- Deterministic success: report includes the correct finding type `secret_exposure`, includes ≥1 correct evidence ID, and remediation mentions rotation + removal.  
-- Intermediate rewards: `search_repo()` surfaces the evidence ID; `create_finding()` with correct type gets partial credit; `validate_finding()` confirms.  
-- False-positive check: claiming *additional* vulnerabilities not verified triggers penalty.
-
-**Medium: Missing security headers**  
-- Goal: detect missing/weak security headers in a service “header snapshot”; propose remediation.  
-- Deterministic success: correct missing header set identification (from a fixed list), plus remediation mapping (e.g., add HSTS, CSP) within the environment’s rubric.  
-- Intermediate rewards: correct tool usage (`check_security_headers()`), correct mapping to finding type, successful verifier validation.  
-- Generalisation: header ordering/extra benign headers vary by seed.
-
-**Hard: Authorisation boundary misconfiguration**  
-- Goal: detect an access control policy bug in a route/role matrix (modelled safely, without exploitation).  
-- Deterministic success: evidence IDs must show the policy mismatch; report must describe impact and remediation (principle of least privilege + policy fix + regression test).  
-- Intermediate rewards: `list_assets()` + `get_log_events()` reveal the mismatch pattern; candidate finding validated.  
-- False-positive guardrail: generic “SQLi/RCE” claims penalised unless evidence supports (it won’t, by design).
-
-**Stretch tasks (post-V1, not for hackathon critical path).**  
-[PROPOSAL] Dependency-risk identification (synthetic CVE mapping), error-handling info leak, prioritisation under strict budget, and multi-finding episodes (2 findings) — but only once the validator-safe V1 is shipped.
-
-## OpenEnv compliance blueprint and repo plan
-
-**SECTION 6 — OpenEnv Compliance Blueprint**
-
-[SOURCED] OpenEnv’s core contract is Gymnasium-like APIs (`reset()`, `step()`, `state()`), with type-safe models, packaged behind a FastAPI server and typically accessed via an EnvClient. citeturn18view0turn19search1  
-
-[SOURCED] For environment creators, OpenEnv explicitly supports `openenv init`, and documents a canonical structure: `models.py`, `client.py`, `server/app.py`, `server/<environment>.py`, plus `openenv.yaml` and packaging metadata. citeturn18view0turn18view1  
-
-[SOURCED] OpenEnv provides CLI commands including `openenv init` and `openenv push` for deploying to **Hugging Face Spaces**. citeturn18view0turn17view0  
-
-[SOURCED] The OpenEnv repo’s environment-building guide demonstrates typed models (Action/Observation/State) as Python dataclasses and a `create_fastapi_app(...)` helper to serve the environment. citeturn19search1  
-
-[SOURCED] The OpenEnv repo explicitly warns *not* to copy outdated manifest patterns; current examples use `spec_version`, `type`, `runtime`, `app`, `port`. citeturn19search2turn23view0  
-
-**Validator-sensitive details you must implement (non-negotiable).**  
-[PROPOSAL] Based on official requirements + observed validator behaviour:  
-- Provide `openenv.yaml` with `spec_version: 1`, `name`, `runtime: fastapi`, `app: server.app:app`, `port: <int>`, and a `tasks:` list with **≥3 tasks each having `id`, `description`, `grader`**. citeturn23view0turn19search2  
-- Ensure each task’s final score is **strictly within (0,1)** to avoid fail-fast validation errors. citeturn27search0turn26view0  
-- Implement an `inference.py` that prints `[START]/[STEP]/[END]` lines exactly and uses the OpenAI SDK for LLM calls (if any), reading `HF_TOKEN`, `API_BASE_URL`, `MODEL_NAME`. citeturn3view6turn22view1turn22view2  
-- Provide a `/health` endpoint that returns 200 once ready (commonly used in examples and deployment docs). citeturn17view0turn20view0  
-
-**Sync vs async.**  
-[SOURCED] OpenEnv supports async-first clients with a `.sync()` wrapper for synchronous usage. For hackathon inference scripts, synchronous control flow is often simpler and widely used in examples. citeturn18view0turn22view4  
-
-**What not to copy from older examples.**  
-[SOURCED] Some course material shows a simplified `openenv.yaml` (`name/version/description`), but the repo’s skill guidance explicitly warns against outdated manifests; follow the current spec-style manifest used in validated examples. citeturn19search2turn19search11turn23view0  
-
-**SECTION 7 — Repo / File Tree Plan**
-
-[SOURCED] OpenEnv’s scaffold and common community submissions converge on a predictable repository layout and file naming. citeturn18view0turn20view0turn23view0  
-
-[PROPOSAL] Recommended repo structure (submission-ready):
-
-```
-secops_evidence_gym/
-  openenv.yaml                 # REQUIRED: spec_version, runtime, app, port, tasks+graders
-  pyproject.toml               # REQUIRED: package metadata + deps
-  README.md                    # REQUIRED: judging narrative + quickstart + safety boundaries
-  inference.py                 # REQUIRED: strict stdout logs + OpenAI client usage
-  models.py                    # REQUIRED: typed Action/Observation/State dataclasses
-  client.py                    # REQUIRED: EnvClient wrapper (sync + async)
-  __init__.py                  # REQUIRED: export Env + models for pip install
-
-  server/
-    app.py                     # REQUIRED: create_fastapi_app(...) wiring + /health
-    environment.py             # REQUIRED: SecOpsEvidenceGymEnvironment(reset/step/state)
-    graders.py                 # REQUIRED: grade_easy/medium/hard + safe_reward clamp
-    tasks.py                   # OPTIONAL (high-leverage): scenario registry + seed sampling
-    safety.py                  # OPTIONAL (high-leverage): tool allowlist + sanitisation helpers
-    requirements.txt           # OPTIONAL (if Docker build uses it)
-    Dockerfile                 # REQUIRED (practically): HF Spaces docker build
-
-  tests/
-    test_api_contract.py       # smoke: reset/step/state doesn’t crash; reward range
-    test_graders.py            # unit: deterministic scoring + strict (0,1) clamp
-    test_seed_determinism.py   # unit: same seed → same evidence IDs
-```
-
-[PROPOSAL] Mandatory for hackathon success: `openenv.yaml`, server app wiring, three tasks+graders, Docker build success, `inference.py` with strict logs, and a README that makes the environment’s value obvious in <60 seconds.
-
-## Reward, grading, and anti-hallucination design
-
-**SECTION 10 — Reward Design**
-
-[SOURCED] OpenEnv leaves reward semantics to the environment; you are responsible for correctness scoring and determinism. citeturn18view0turn19search1  
-
-[SOURCED] Hackathon validation has shown strict “score must be between 0 and 1 (not 0.0 and not 1.0)” behaviour, and teams clamp rewards (e.g., 0.01–0.99). citeturn27search0turn26view0  
-
-[SOURCED] Empirical RL research in other domains (e.g., autonomous racing) shows reward design choices materially affect performance and generalisation, supporting the need for careful shaping rather than a single sparse terminal reward. citeturn15view2  
-
-[PROPOSAL] **Core principle:** correctness is **verifier-gated**, not language-judged. You can optionally add *format/style* checks, but never allow style to dominate correctness reward.
-
-### Reward structure (practical V1)
-
-[PROPOSAL] Normalise the final *task score* into `(0,1)` and keep per-step rewards small enough that summed episode reward stays in `(0,1)` as well (or only final reward is used, depending on your environment semantics). Use a single “score” to satisfy the validator and expose detailed breakdowns in `observation.metadata`.
-
-**Terminal (sparse) components** ✅  
-[PROPOSAL]  
-- `+0.60` if at least one ground-truth finding is verified and correctly described (type + impact).  
-- `+0.15` if the report includes **≥1 valid evidence ID** per finding and those IDs correspond to the right artefacts.  
-- `+0.15` if remediation is actionable (specific control, config, test).  
-- `-0.40` per hallucinated/unverified finding claimed in the report.  
-- `-0.20` if the agent fails to run `validate_finding()` before `submit_report()`.
-
-**Intermediate (dense) components** 🧭  
-[PROPOSAL]  
-- `+0.02` for discovering a *new* relevant evidence ID (first time only).  
-- `+0.03` for creating a well-formed candidate finding that references evidence IDs.  
-- `-0.01` per step (efficiency pressure).  
-- `-0.03` for repeating the same tool call (exact same args) beyond 2 times.  
-
-**False-positive penalties / anti-hallucination** 🧯  
-[PROPOSAL] A “hallucination” is operationally defined as: the report asserts a finding that is not in the environment’s `verified_findings` list. This is easy to compute deterministically and maps directly to your stated goal (“avoid hallucinating findings”).
-
-### Avoiding reward hacking
-
-[PROPOSAL] Hardening rules:  
-- Cap rewards from verbosity: extra words do not add points.  
-- Make evidence IDs required for high scores (prevents purely rhetorical “security speak”).  
-- Penalise calling `validate_finding()` repeatedly without new evidence.  
-- Reject “kitchen sink” reporting by penalising extra unverified findings.
-
-### Binary vs shaped reward
-
-[PROPOSAL] **Binary-only** (0/1) will be easy to implement but brittle for multi-step tool use; the agent gets no gradient for *how* to investigate efficiently.  
-
-[PROPOSAL] **Lightly shaped** (recommended) keeps correctness deterministic while providing enough signal to train investigation workflow (evidence collection, validation order, loop avoidance). This mirrors the broader lesson from reward engineering research: shaping and tuning can significantly alter learning outcomes. citeturn15view2  
-
-### Deterministic judge vs hybrid judge
-
-[PROPOSAL]  
-- **Strict deterministic judge (recommended V1):** all correctness via verifiers + string/structure checks.  
-- **Hybrid (stretch):** add a small LLM-based style score (e.g., clarity), heavily downweighted (≤0.05 of total) and never affecting pass/fail correctness.
-
-## Baseline inference pipeline and strict stdout logging
-
-**SECTION 11 — Baseline Inference Pipeline**
-
-[SOURCED] Hackathon requirements include: a reproducible `inference.py`, the OpenAI client requirement for LLM calls (using provided env vars), and strict stdout logging. citeturn3view6  
-
-[SOURCED] A concrete, hackathon-aligned stdout format has been used by validated submissions (example):  
-- `[START] task=<name> env=<benchmark> model=<model_name>`  
-- `[STEP] step=<n> action=<str> reward=<0.00> done=<true|false> error=<msg|null>`  
-- `[END] task=<name> success=<true|false> steps=<n> score=<0.00> rewards=<r1,r2,...>` citeturn22view1turn22view2  
-
-[SOURCED] The same example inference uses the OpenAI SDK, reading `API_BASE_URL`, `MODEL_NAME`, and `HF_TOKEN`. citeturn22view1turn22view4  
-
-### Responsibilities of `inference.py`
-
-[PROPOSAL] `inference.py` should:  
-- read env vars: `HF_TOKEN`, `API_BASE_URL`, `MODEL_NAME`, `ENV_URL` (and optionally `TASK_NAME` override),  
-- connect to the env via `.sync()` client,  
-- run tasks in a fixed order (easy → medium → hard),  
-- execute a bounded number of steps per task,  
-- log exactly one `[START]...` per task, one `[END]...` per task, and a `[STEP]...` per environment step,  
-- always exit with code 0 (even on failures) and log errors in the `[STEP] error=` field to avoid hard crashes.
-
-### Control flow (V1 baseline strategy)
-
-[PROPOSAL] Use a **hybrid baseline** that is reliable under time constraints:  
-- scripted tool sequence per task (fast, deterministic),  
-- one LLM call (optional) to draft the final report from gathered evidence (so the demo shows “agentic reasoning”),  
-- temperature fixed to 0 for reproducibility (and lower variance).  
-
-[SOURCED] Deterministic inference settings like `TEMPERATURE=0.0` are used in competitive OpenEnv hackathon baselines. citeturn20view0turn22view4  
-
-### Minimum viable baseline (must ship)
-
-[PROPOSAL] For each task:  
-1) `reset(task_id=<tier>)`  
-2) run 2–4 tool calls that are always relevant (e.g., `check_security_headers`, `search_repo`, etc.)  
-3) `create_finding(...)` using evidence IDs  
-4) `validate_finding(finding_id)`  
-5) `submit_report(report_json)`  
-
-### Stronger baseline (only if time permits)
-
-[PROPOSAL] Add one planning LLM call that chooses among tools based on the alert type, but still keep a hard step limit, and always include verifier validation before reporting.
-
-## Complete build, validation, deployment, and submission pipeline
-
-**SECTION 5 — Complete End-to-End Pipeline**
-
-[SOURCED] This pipeline is built to satisfy both OpenEnv conventions (init/push, typed models, FastAPI server) and hackathon validation constraints (tasks/graders, inference logging, runtime budgets). citeturn18view0turn19search2turn3view6turn22view1  
-
-### Phase goals, deliverables, verification (execution-ready)
-
-[PROPOSAL] The table below is the “do-this-in-order” execution plan. It is intentionally validator-first.
-
-| Phase | Goal | Deliverables | Files touched | Acceptance criteria | Main risks | How to verify |
-|---|---|---|---|---|---|---|
-| Scope lock | Freeze V1 to 3 tasks + bounded tools | 1-page spec + non-goals | README.md | No pentest/exploit scope; 3 tasks defined | Scope creep | Manual checklist |
-| Scaffold | Generate OpenEnv skeleton | Working importable package | openenv.yaml, models.py, client.py, server/* | `python -c "import ..."` succeeds | Wrong template/paths | Local import smoke test |
-| Environment core | Implement reset/step/state; tool router | Simulator runs end-to-end | server/environment.py | reset+step returns typed observation; no crashes | Action validation crashes | manual `curl` + python client |
-| Tasks + graders | Implement 3 graders + strict (0,1) clamp | `grade_easy/medium/hard` | server/graders.py, openenv.yaml | tasks discoverable; scores strictly in (0,1) | Validator fail-fast | unit tests + manual checks |
-| Baseline inference | Make inference reproducible + strict logs | inference.py | inference.py | prints correct `[START]/[STEP]/[END]` | log-parser failure | run script locally |
-| Local validation | Run OpenEnv build & validate | passes `openenv validate` | Dockerfile, server/app.py | validate passes locally | port mismatch | `openenv validate --url ...` |
-| Docker + HF | Deploy to Spaces | live endpoint | openenv push output | `/health` 200; reset+step works remotely | HF port/env mismatch | curl + python client |
-| Submission | Final narrative + demo | polished README + screenshots | README.md | demo works in <2 min | unclear story | run “demo script” |
-
-### Concrete build plan with commands
-
-[SOURCED] OpenEnv supports `openenv init` and `openenv push` and documents this as the standard creator workflow. citeturn18view0turn17view0  
-[SOURCED] The OpenEnv course also provides a grounded dev loop: `uv sync`, `uv run server`, `curl /health`, and Docker build/run commands. citeturn17view0  
-
-[PROPOSAL] Commands (copy/paste order):
-
-1) **Scaffold**
-```bash
-pip install openenv-core
-openenv init secops_evidence_gym
-cd secops_evidence_gym
-```
-[SOURCED] `openenv init` is the documented way to scaffold a new environment. citeturn18view0turn18view2  
-
-2) **Local dev install + run**
-```bash
-uv sync
-uv run server
-curl http://localhost:8000/health
-```
-[SOURCED] `uv run server` and `/health` checks are part of the recommended iteration loop in OpenEnv course materials. citeturn17view0  
-
-3) **Implement core files (edit)**
-- `models.py`: define `Action/Observation/State` dataclasses  
-- `server/environment.py`: implement reset/step/state + tool routing  
-- `server/graders.py`: implement `grade_easy/grade_medium/grade_hard` + `safe_reward()`  
-- `openenv.yaml`: add `tasks:` with grader import paths  
-
-[SOURCED] OpenEnv’s environment-building guide explicitly directs you to define models and implement `reset/step/state`, then wire a FastAPI app. citeturn19search1  
-[SOURCED] A validator-aligned `openenv.yaml` with `spec_version`, `runtime`, `app`, `port`, and `tasks` exists in deep-validation passing examples. citeturn23view0  
-
-4) **Build + validate (local)**
-```bash
-openenv build
-openenv validate --verbose
-```
-[SOURCED] `openenv build` and `openenv validate` are part of OpenEnv’s recommended validation workflow. citeturn19search2  
-
-5) **Docker build/run smoke test**
-```bash
-docker build -t secops-evidence-gym:latest -f server/Dockerfile .
-docker run -p 8000:8000 secops-evidence-gym:latest
-curl http://localhost:8000/health
-```
-[SOURCED] This `docker build -f server/Dockerfile .` pattern is directly shown in OpenEnv deployment course material. citeturn17view0  
-
-6) **Run inference locally**
-```bash
-export HF_TOKEN="..."
-export API_BASE_URL="..."
-export MODEL_NAME="..."
-export ENV_URL="http://localhost:8000"
-python inference.py
-```
-[SOURCED] These env var names and OpenAI SDK usage are consistent with hackathon guidance and existing inference implementations. citeturn3view6turn22view4  
-
-7) **Deploy to Hugging Face Spaces**
-```bash
-openenv push --repo-id <your-hf-username>/secops-evidence-gym
-```
-[SOURCED] `openenv push` is described as the fastest path to deploy to **Hugging Face Spaces**. citeturn17view0turn18view0  
-
-### Testing and validation plan (high-signal)
-
-[SOURCED] OpenEnv stresses predictable API behaviour and type-safe contracts; hackathon validation is fail-fast. citeturn18view0turn27search0  
-
-[PROPOSAL] Test layers (in priority order):  
-- **API contract smoke tests:** reset/step/state return valid JSON; never crash on invalid tool name (should return an observation with an error field).  
-- **Grader tests:** for each task, verify (a) correctness cases score high, (b) hallucination cases score low, (c) score always ∈ (0,1).  
-- **Seed determinism tests:** same `seed` produces same evidence IDs and same verifier outputs.  
-- **Runtime test:** run `inference.py` end-to-end and assert wall-clock < 2 minutes locally; assume < 20 minutes on grader infra even with cold starts. citeturn3view6turn22view4  
-- **Reward sanity tests:** ensure reward increases monotonically with verified correctness; fails if verbosity alone increases reward.
-
-## Submission packaging, execution roadmap, real-world usefulness, and failure modes
-
-**SECTION 14 — README / Demo / Submission Narrative**  
-[SOURCED] Judges likely assess both the environment’s technical correctness (programmatic checks) and qualitative merit (LLM scoring / narrative). citeturn3view7  
-
-[PROPOSAL] README structure that “feels like a winner” 🏆:  
-- **Hero block:** one-paragraph pitch + why it’s real-world + safety claim.  
-- **Two-minute demo:** copy/paste commands + expected output snippet with `[START]/[STEP]/[END]`.  
-- **Environment contract:** action schema, observation schema, task list.  
-- **Grading:** explain deterministic verifiers + hallucination penalties.  
-- **Safety & isolation:** explicit exclusions (no egress, no shell, synthetic artefacts).  
-- **Real-world relevance:** how this benchmarks/reporting maps to security workflows (triage, evidence, remediation).  
-- **Screenshots:** web UI (optional) + an evidence trace + one scored report example.  
-
-**SECTION 15 — Project Management Plan**  
-[PROPOSAL] Day-by-day (assuming a hackathon-style sprint):
-
-- **Day 0 (scope lock + scaffold):** environment skeleton, `openenv.yaml` with 3 tasks, stub graders returning 0.5 (clamped), server runs locally.  
-- **Day 1 (determinism + validator):** implement scenario generator, evidence registry, verifiers, and strict (0,1) scoring; pass `openenv validate`.  
-- **Day 2 (baseline + polish):** implement `inference.py` strict logs; deploy to Spaces; polish README + demo artefacts.
-
-[PROPOSAL] Critical path: `openenv.yaml tasks+graders` → grader clamp `(0,1)` → inference stdout format → Docker+Spaces deployment. (Everything else is secondary.)
-
-**SECTION 16 — Real-World Usefulness Plan**  
-[SOURCED] NIST’s testing guide emphasises planning, conducting tests, analysing findings, and developing mitigation strategies; your environment’s “evidence → remediation” focus aligns with that lifecycle without requiring offensive exploitation. citeturn29search8turn29search0  
-
-[PROPOSAL] Who would care after the hackathon:  
-- security engineering teams evaluating agentic “triage + reporting” reliability,  
-- LLM tooling teams wanting benchmarks for **non-hallucinating, evidence-grounded** outputs,  
-- training teams building safe cyber ranges (without weaponisation).
-
-[PROPOSAL] Post-hackathon upgrades (highest leverage):  
-- export trajectories as JSONL for offline training,  
-- add more scenario families (still safe) and a held-out split for generalisation,  
-- integrate with RL trainers (e.g., TRL’s OpenEnv integration) to show real training curves. citeturn19search6turn10view0  
-
-[SOURCED] PenGym provides evidence that realism/faithfulness of environments can affect transfer and stability when moving from simulation to more realistic settings—so you should roadmap a “higher fidelity mode” (still safe) later, not in V1. citeturn15view0  
-
-**SECTION 17 — Why the naive version would fail**  
-[PROPOSAL] Top failure patterns (and why they kill submissions):  
-- Too broad (full cyber range, live services): fails time/infra constraints. citeturn3view6turn10view0  
-- Fuzzy grading (LLM-only judging): non-deterministic, easy to game.  
-- Unbounded tools (shell/network): unsafe + untrainable action space.  
-- Scores at exactly 0.0 or 1.0: fail-fast “out of range” validator. citeturn27search0turn26view0  
-- Inference logs not parseable: phase-1 failure even if env is good. citeturn3view6turn22view1  
-- Port / health issues on Spaces: container “works locally” but fails remotely. citeturn17view0turn20view0  
-
-**SECTION 18 — Final Recommendation**
-
-[PROPOSAL] **What should you build?**  
-Build **SecOps Evidence Gym**: a deterministic, safe, sandbox-only cyber analyst environment focused on evidence collection, verifier validation, and remediation reporting.
-
-[PROPOSAL] **What should V1 include? (minimum winning set)**  
-- OpenEnv-compliant FastAPI env with typed models and `reset/step/state`. citeturn18view0turn19search1  
-- `openenv.yaml` with **3 tasks + graders**. citeturn23view0turn3view6  
-- Deterministic verifiers + strict score clamp to `(0,1)`. citeturn27search0turn26view0  
-- Baseline `inference.py` with strict `[START]/[STEP]/[END]` logging + OpenAI SDK usage for any LLM calls. citeturn3view6turn22view1turn22view4  
-- HF Spaces deployment with a working `/health`. citeturn17view0turn20view0  
-
-[PROPOSAL] **What should you cut?**  
-- Any real pentesting/offensive content, any arbitrary command execution, any live targets, any correctness scoring via an LLM judge.
-
-[PROPOSAL] **Top 5 implementation decisions that matter most**  
-1) Validator-safe `openenv.yaml` tasks+graders wiring. citeturn23view0  
-2) Score/range compliance: clamp to `(0,1)` everywhere. citeturn27search0turn26view0  
-3) Strict stdout format in `inference.py`. citeturn22view1turn22view2  
-4) Deterministic verifiers as the source of truth.  
-5) Bounded tool set (≤8 tools) with anti-loop penalties.
-
-[PROPOSAL] **Minimum viable winning submission**  
-A V1 with 3 tasks, deterministic graders, bounded tools, strict inference logging, and a polished README + demo trace.
-
-[PROPOSAL] **Minimum viable real-world useful submission**  
-The same V1, plus: seed determinism, trajectory export, and a clear “how to add new scenarios” contributor guide.
-
-[PROPOSAL] **If you only have time for 20% of ambition—do this exact 20%:**  
-- Implement **one** robust multi-step loop (tools → validate → report)  
-- Implement **exactly 3** tasks (easy/medium/hard)  
-- Make graders deterministic and validator-safe  
-- Make deployment + inference bulletproof  
-Everything else is stretch.
-
-**Confidence (my estimate): 8.4/10** ✅🔥
-
-## Sources and credibility ratings (with exact links)
-
-[SOURCED] Ratings are my judgement of authority + relevance for this hackathon context (0–10). URLs are provided verbatim in code form.
-
-### Tier 1 (official OpenEnv + hackathon dashboard)
-- Credibility **9.5/10** — `https://github.com/meta-pytorch/OpenEnv` citeturn18view0  
-- Credibility **9.0/10** — `https://github.com/meta-pytorch/OpenEnv/blob/main/envs/README.md` citeturn19search1  
-- Credibility **8.5/10** — `https://github.com/meta-pytorch/OpenEnv/blob/main/.claude/skills/generate-openenv-env/SKILL.md` citeturn19search2  
-- Credibility **9.0/10** — `https://www.scaler.com/school-of-technology/meta-pytorch-hackathon/dashboard` citeturn1view0turn3view6turn3view7  
-
-### Tier 2 (strong community exemplars)
-- Credibility **8.5/10** — `https://github.com/sid-rp/kube-sre-gym` citeturn10view0  
-- Credibility **8.0/10** — `https://huggingface.co/openenv-community` citeturn14view0  
-- Credibility **7.5/10** — `https://github.com/Harikishanth/Incident-Triage-Environment` citeturn20view0turn23view0turn22view1  
-
-### Tier 3 (peer-reviewed / primary references for design constraints)
-- Credibility **8.5/10** — PenGym (Computers & Security, open access): `https://www.sciencedirect.com/science/article/pii/S0167404824004450` citeturn15view0  
-- Credibility **8.0/10** — Reward design + generalisation (Scientific Reports, 2025): `https://www.nature.com/articles/s41598-025-27702-6` citeturn15view2  
-- Credibility **8.5/10** — AMaze (JOSS, 2025): `https://joss.theoj.org/papers/10.21105/joss.07208` citeturn16search7  
-- Credibility **9.5/10** — NIST SP 800-115: `https://csrc.nist.gov/pubs/sp/800/115/final` citeturn29search8  
-- Credibility **9.0/10** — NIST “Cyber Range: A Guide” (PDF landing): `https://www.nist.gov/document/cyber-range` citeturn29search1  
-- Credibility **7.5/10** — “Cybersecurity of Cyber Ranges: Threats and Mitigations” (IJISR, 2022 PDF): `https://infonomics-society.org/wp-content/uploads/Cybersecurity-of-Cyber-Ranges.pdf` citeturn29search2  
-
-### Tier 4 (useful validator “ground truth” signals from the field)
-- Credibility **6.5/10** — Validator failure mode discussion (score must be strictly between 0 and 1): `https://www.reddit.com/r/pytorch/comments/1shi767/meta_x_pytorch_x_sst_x_openenv_hackathon_phase_2/` citeturn27search0  
-- Credibility **7.0/10** — Strict logging format reference via a verified submission’s `inference.py`: `https://github.com/Harikishanth/Incident-Triage-Environment/blob/main/inference.py` citeturn22view1turn22view2  
-
-### Uploaded reference you provided
+# OpenEnv Hackathon Execution Pipeline for a Safe Cybersecurity Analyst Environment
+
+## Executive decision and scope selection
+
+**SECTION 1 — Executive Decision**
+
+[SOURCED] The hackathon’s *validator + judging* constraints strongly favour environments that: (a) simulate a real-world task (not “games/toys”), (b) are fully OpenEnv-compliant, (c) ship with **≥3 tasks with graders**, (d) produce **scores/rewards in the 0–1 range**, (e) include a reproducible `inference.py` that uses the **OpenAI client** (for any LLM calls) and prints strict `[START]/[STEP]/[END]` logs, and (f) run within a ~**20 minute** budget on ~**2 vCPU / 8 GB** infra. citeturn3view6turn3view7turn22view1turn22view2  
+
+[INFERENCE] Under these realities, your cybersecurity-analyst direction is *not* automatically the best, but it *can* become a high-probability-to-win choice if—and only if—you narrow to a deterministic, bounded, “investigate → cite evidence → verify → remediate” loop where (i) tools are tightly sandboxed, (ii) graders are deterministic, and (iii) the action space is small enough to be learnable and demo-able under the runtime cap.
+
+[PROPOSAL] **Decision:** keep the cybersecurity direction, but **narrow aggressively** to a V1 environment that benchmarks **disciplined security triage + evidence-grounded reporting**, not pentesting/exploitation. The V1 I recommend building is:
+
+[PROPOSAL] **“SecOps Evidence Gym”** — a safe, isolated OpenEnv environment where an agent investigates a *synthetic* microservice “organisation” via a **bounded tool API**, collects **evidence IDs**, validates candidate findings through **deterministic verifiers**, and submits a structured remediation report.  
+
+[SOURCED] This matches strong “winner DNA” seen in `kube-sre-gym` (realistic professional workflow + verification + narrative clarity) while remaining implementable in a hackathon budget. citeturn10view0turn18view0  
+
+[PROPOSAL] **What to cut entirely in V1 (non-negotiable):**  
+- “Live target” behaviour; no external network targets; no arbitrary HTTP to the internet. 🔒  
+- Any exploit payload recipes, exploit chains, privilege-escalation playbooks, or “how to hack X”. 🔒  
+- Arbitrary shell access (`bash`, `kubectl`, `nmap`, etc.) inside the environment. (Action space explosion + safety risk.)  
+- LLM-only grading/judging for correctness. (Reward hacking + non-determinism.)  
+
+[SOURCED] **What to keep (but narrow):** tool-using investigation, multi-step interaction, and deterministic verification—these are consistent with what OpenEnv is designed to support (typed `reset/step/state`, isolated server, type-safe schemas). citeturn18view0turn19search1  
+
+**SECTION 3 — Candidate Scope Comparison**
+
+[SOURCED] The scoring below is anchored on hackathon validator requirements (3+ graded tasks, 0–1 scoring, strict inference logging, runtime limits) plus OpenEnv’s scaffolding/CLI/deployment model. citeturn3view6turn18view0turn22view1  
+
+[PROPOSAL] Weighted criteria (sum=1.00): judging fit 0.14, OpenEnv fit 0.12, grader determinism 0.14, implementation risk 0.12, runtime feasibility 0.10, demoability 0.10, real-world usefulness 0.10, novelty 0.08, training usefulness 0.06, shipping-on-time likelihood 0.04.
+
+| Candidate scope | Judging fit | OpenEnv fit | Determinism | Impl risk (lower=better) | Runtime | Demo | Real-world use | Novelty | Training use | Ship-on-time | Weighted total |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| **A. Your original direction:** “disciplined cyber analyst investigating a sandbox” (broad) | 8 | 7 | 6 | 4 | 6 | 8 | 8 | 8 | 8 | 4 | **6.7** |
+| **B. Narrow cyber variant (recommended):** evidence-first triage lab with bounded tools + deterministic verifiers + structured report | 9 | 8 | 9 | 7 | 9 | 9 | 8 | 7 | 8 | 8 | **8.4** |
+| **C. Adjacent: SRE incident triage (single-turn, deterministic logs → RCA)** | 9 | 8 | 10 | 9 | 10 | 8 | 8 | 5 | 6 | 9 | **8.3** |
+| **D. Adjacent: prompt-injection “WAF” benchmark** | 7 | 8 | 8 | 7 | 9 | 9 | 7 | 7 | 6 | 7 | **7.6** |
+
+[INFERENCE] Candidate C (SRE triage) is extremely validator-safe (many examples already pass deep validation), but it is likely more saturated and less “new” in judging. Candidate B keeps your cybersecurity theme while retaining the determinism and boundedness that the validator and time budget demand, so it is the best balance for **winning + real-world usefulness**.
+
+**SECTION 4 — Final V1 Problem Statement**
+
+[PROPOSAL] **One-sentence version:** Build a safe OpenEnv environment that trains and benchmarks an agent to perform **evidence-grounded security triage** on a bounded synthetic system and produce a **remediation-oriented report** without hallucinating.
+
+[PROPOSAL] **Short pitch (demo-ready):** “SecOps Evidence Gym” gives the model an alert and a constrained set of investigation tools. The model must gather evidence, validate findings with deterministic verifiers, and submit a structured report. Scores reward verified correctness, penalise hallucinated claims and wasted steps, and remain strictly within (0,1) to satisfy the hackathon validator. ✅🔒
+
+[PROPOSAL] **Precise implementation version:** A FastAPI OpenEnv server exposing typed `reset()`, `step()`, `state()` and a manifest `openenv.yaml` with at least three tasks (easy/medium/hard), each associated with a grader. The environment implements multi-step tool calls inside `step()`, and uses deterministic verifiers plus a strict score clamp to `(0,1)` for validator compatibility. citeturn18view0turn19search1turn23view0turn26view0turn27search0  
+
+## Winner patterns and judging fit extraction
+
+**SECTION 2 — Winner Pattern Extraction**
+
+[SOURCED] `kube-sre-gym` (OpenEnv Hackathon SF winner) demonstrates several patterns that appear strongly aligned with what judges value: a **realistic professional task**, an explicit **multi-step workflow** (triage → investigate → fix → verify), **multi-layer verification** (programmatic health checks + judge), a strong narrative that explains learning dynamics, and deployment on **Hugging Face Spaces**. citeturn10view0turn14view0  
+
+image_group{"layout":"carousel","aspect_ratio":"16:9","query":["kube-sre-gym OpenEnv Hackathon winner screenshot","OpenEnv framework architecture diagram","Hugging Face Spaces OpenEnv environment web demo","Incident Triage Environment OpenEnv Hackathon 2026 screenshot"],"num_per_query":1}
+
+[SOURCED] Concretely, `kube-sre-gym` highlights: “real cluster/tool interaction”, verification layers to prevent false success, curriculum progression, and strong documentation that makes the environment’s value obvious quickly. citeturn10view0  
+
+[SOURCED] A 2026 hackathon submission that explicitly claims Phase-2 deep-validation success (`Incident-Triage-Environment`) reveals particularly transferable “validator-winning” patterns:  
+- A manifest with `spec_version`, `runtime`, `app`, `port`, and a **`tasks:` list where each task has a `grader:` pointing to importable Python functions**. citeturn23view0  
+- Deterministic graders that clamp outputs to a validator-friendly range. citeturn26view0turn26view3  
+- An `inference.py` that uses the OpenAI SDK and prints the strict stdout protocol with `[START]`, `[STEP]`, `[END]` lines in a stable key=value format. citeturn22view1turn22view2turn22view4  
+
+[SOURCED] The official OpenEnv repo reinforces what is transferable: type-safe action/observation/state contracts, containerised isolation, and standard Gym-like APIs. It also explicitly says OpenEnv is experimental and APIs can change, which increases the value of a minimal, validator-first build loop. citeturn18view0turn19search2  
+
+[INFERENCE] Given hackathon evaluation combines programmatic checks with LLM scoring, you must optimise for **deterministic correctness** *and* a compelling narrative/demo. citeturn3view7turn3view6  
+
+[PROPOSAL] Transferable “winner patterns” you should copy (selectively):  
+- **Strong “professional workflow” framing** (SRE, security analyst, triage) with clear step boundaries.  
+- **Small, discoverable tool set** that mirrors real practice (logs, config, policy checks) while staying bounded.  
+- **Deterministic verification** (programmatic checks) as the source of truth for correctness.  
+- **Narrative traceability**: logs, episode IDs, and a short “watch the agent work” demo.  
+- **Deployment excellence**: clean Docker build, working `/health`, working `/web` UI if enabled, and reproducible inference.
+
+[PROPOSAL] What *not* to copy blindly:  
+- The “real cluster” dependency (e.g., GKE) is high operational burden and can fail the hackathon’s limited infra budget. citeturn10view0turn3view6  
+- LLM-as-judge for correctness (too easy to reward-hack; non-deterministic). (Use it, at most, for *format/style*, not correctness.)  
+
+## Core V1 environment design and benchmark tasks
+
+**SECTION 8 — Core Environment Design**
+
+**V1 concept (aggressively narrow).**  
+[PROPOSAL] Your environment is a **synthetic organisation** with a small, fixed topology (three “services” + artefacts). The agent receives an alert. It can only interact via **approved tools** (implemented inside the simulator). It must (a) gather evidence IDs, (b) validate candidate findings, and (c) submit a report.
+
+**Topology (V1).**  
+[PROPOSAL] Fixed components (no containers inside containers):  
+- `gateway` (public entry), `profile-service`, `admin-service`  
+- `repo_snapshot` (static code/config excerpts)  
+- `telemetry` (sanitised logs + “header snapshot” + “dependency manifest snapshot”)  
+
+**Reset logic.**  
+[PROPOSAL] `reset(task_id=..., seed=...)` selects a scenario variant and initialises:  
+- episode ID, step count  
+- scenario ground truth (one injected issue per episode in V1)  
+- tool budgets + “allowed scope” banner  
+- an evidence registry mapping `EVID-### → artefact snippet`  
+Return an initial observation containing the alert, the tool catalogue, and an empty “verified findings” list.
+
+**Randomisation strategy.**  
+[PROPOSAL] Use seed-driven, deterministic randomisation:  
+- rename services/routes/IDs (`profile-service` might become `user-profile`),  
+- shuffle benign log lines around the key evidence,  
+- vary exact header sets / dependency versions within a small closed set,  
+- keep each scenario **fully reproducible from the seed**.
+
+[SOURCED] Benchmark generators (e.g., AMaze) exist specifically to create diverse but controlled environments for evaluating generalisation, supporting the idea of seeded procedural variation rather than a single static scenario. citeturn16search7turn16search1  
+
+**Safety boundaries.**  
+[PROPOSAL] The sandbox contains **no live targets**, no real secrets, and no outbound network. “Secrets” are synthetic strings with an explicit “DO NOT USE OUTSIDE LAB” marker. Tools return synthetic results only. 🔒  
+
+[SOURCED] NIST’s cyber range guidance emphasises cyber ranges as safe and legal environments for training and assessment; separate research also discusses that cyber ranges themselves have security risks that must be mitigated (e.g., leakage/misuse), reinforcing the need for strict isolation and artefact sanitisation. citeturn29search1turn29search2  
+
+**How state is exposed to the agent.**  
+[PROPOSAL] Expose only a concise state summary: current phase, step budget remaining, tools remaining, verified findings count, and recent evidence IDs. Keep full ground truth hidden.
+
+**Tool/action design (bounded action space).**  
+[PROPOSAL] V1 tool list (keep it ≤8 tools):  
+1) `list_assets()` → returns asset IDs and route IDs  
+2) `get_log_events(service_id, query)` → returns evidence IDs  
+3) `check_security_headers(service_id)` → returns evidence IDs + pass/fail list  
+4) `search_repo(query)` → returns evidence IDs from code snippets  
+5) `scan_dependencies()` → returns evidence IDs from a lockfile excerpt  
+6) `create_finding(finding_type, evidence_ids, severity_guess, remediation)` → stores candidate finding  
+7) `validate_finding(finding_id)` → deterministic verifier; returns `(verified, matching_gt_id)`  
+8) `submit_report(report_json)` → terminal action  
+
+**Anti-loop logic.**  
+[PROPOSAL] Track action signatures `(tool_name, args_hash)` and:  
+- apply increasing penalties for repeats,  
+- hard-stop an episode if identical actions repeat ≥6 times, returning `done=True` with a low score,  
+- always return a valid observation (never a server crash) to preserve training rollouts.
+
+[SOURCED] OpenEnv’s environment-creation guidance strongly implies you should implement robust behaviour around `reset/step/state` with typed contracts and predictable server behaviour. citeturn19search1turn18view0  
+
+**SECTION 9 — Tasks / Benchmarks**
+
+[SOURCED] The hackathon requires **at least 3 tasks with graders** and explicitly checks the tasks registry. citeturn3view6turn27search0  
+
+[PROPOSAL] V1 ships exactly **3 flagship tasks**, difficulty-tiered, each with deterministic success criteria and intermediate milestones.
+
+**Flagship tasks (easy/medium/hard).**  
+[PROPOSAL] Each task is a *family* with small seeded variants.
+
+**Easy: Secret exposure in repo snapshot**  
+- Goal: identify a leaked synthetic API key in a config file excerpt; propose rotation/removal.  
+- Deterministic success: report includes the correct finding type `secret_exposure`, includes ≥1 correct evidence ID, and remediation mentions rotation + removal.  
+- Intermediate rewards: `search_repo()` surfaces the evidence ID; `create_finding()` with correct type gets partial credit; `validate_finding()` confirms.  
+- False-positive check: claiming *additional* vulnerabilities not verified triggers penalty.
+
+**Medium: Missing security headers**  
+- Goal: detect missing/weak security headers in a service “header snapshot”; propose remediation.  
+- Deterministic success: correct missing header set identification (from a fixed list), plus remediation mapping (e.g., add HSTS, CSP) within the environment’s rubric.  
+- Intermediate rewards: correct tool usage (`check_security_headers()`), correct mapping to finding type, successful verifier validation.  
+- Generalisation: header ordering/extra benign headers vary by seed.
+
+**Hard: Authorisation boundary misconfiguration**  
+- Goal: detect an access control policy bug in a route/role matrix (modelled safely, without exploitation).  
+- Deterministic success: evidence IDs must show the policy mismatch; report must describe impact and remediation (principle of least privilege + policy fix + regression test).  
+- Intermediate rewards: `list_assets()` + `get_log_events()` reveal the mismatch pattern; candidate finding validated.  
+- False-positive guardrail: generic “SQLi/RCE” claims penalised unless evidence supports (it won’t, by design).
+
+**Stretch tasks (post-V1, not for hackathon critical path).**  
+[PROPOSAL] Dependency-risk identification (synthetic CVE mapping), error-handling info leak, prioritisation under strict budget, and multi-finding episodes (2 findings) — but only once the validator-safe V1 is shipped.
+
+## OpenEnv compliance blueprint and repo plan
+
+**SECTION 6 — OpenEnv Compliance Blueprint**
+
+[SOURCED] OpenEnv’s core contract is Gymnasium-like APIs (`reset()`, `step()`, `state()`), with type-safe models, packaged behind a FastAPI server and typically accessed via an EnvClient. citeturn18view0turn19search1  
+
+[SOURCED] For environment creators, OpenEnv explicitly supports `openenv init`, and documents a canonical structure: `models.py`, `client.py`, `server/app.py`, `server/<environment>.py`, plus `openenv.yaml` and packaging metadata. citeturn18view0turn18view1  
+
+[SOURCED] OpenEnv provides CLI commands including `openenv init` and `openenv push` for deploying to **Hugging Face Spaces**. citeturn18view0turn17view0  
+
+[SOURCED] The OpenEnv repo’s environment-building guide demonstrates typed models (Action/Observation/State) as Python dataclasses and a `create_fastapi_app(...)` helper to serve the environment. citeturn19search1  
+
+[SOURCED] The OpenEnv repo explicitly warns *not* to copy outdated manifest patterns; current examples use `spec_version`, `type`, `runtime`, `app`, `port`. citeturn19search2turn23view0  
+
+**Validator-sensitive details you must implement (non-negotiable).**  
+[PROPOSAL] Based on official requirements + observed validator behaviour:  
+- Provide `openenv.yaml` with `spec_version: 1`, `name`, `runtime: fastapi`, `app: server.app:app`, `port: <int>`, and a `tasks:` list with **≥3 tasks each having `id`, `description`, `grader`**. citeturn23view0turn19search2  
+- Ensure each task’s final score is **strictly within (0,1)** to avoid fail-fast validation errors. citeturn27search0turn26view0  
+- Implement an `inference.py` that prints `[START]/[STEP]/[END]` lines exactly and uses the OpenAI SDK for LLM calls (if any), reading `HF_TOKEN`, `API_BASE_URL`, `MODEL_NAME`. citeturn3view6turn22view1turn22view2  
+- Provide a `/health` endpoint that returns 200 once ready (commonly used in examples and deployment docs). citeturn17view0turn20view0  
+
+**Sync vs async.**  
+[SOURCED] OpenEnv supports async-first clients with a `.sync()` wrapper for synchronous usage. For hackathon inference scripts, synchronous control flow is often simpler and widely used in examples. citeturn18view0turn22view4  
+
+**What not to copy from older examples.**  
+[SOURCED] Some course material shows a simplified `openenv.yaml` (`name/version/description`), but the repo’s skill guidance explicitly warns against outdated manifests; follow the current spec-style manifest used in validated examples. citeturn19search2turn19search11turn23view0  
+
+**SECTION 7 — Repo / File Tree Plan**
+
+[SOURCED] OpenEnv’s scaffold and common community submissions converge on a predictable repository layout and file naming. citeturn18view0turn20view0turn23view0  
+
+[PROPOSAL] Recommended repo structure (submission-ready):
+
+```
+secops_evidence_gym/
+  openenv.yaml                 # REQUIRED: spec_version, runtime, app, port, tasks+graders
+  pyproject.toml               # REQUIRED: package metadata + deps
+  README.md                    # REQUIRED: judging narrative + quickstart + safety boundaries
+  inference.py                 # REQUIRED: strict stdout logs + OpenAI client usage
+  models.py                    # REQUIRED: typed Action/Observation/State dataclasses
+  client.py                    # REQUIRED: EnvClient wrapper (sync + async)
+  __init__.py                  # REQUIRED: export Env + models for pip install
+
+  server/
+    app.py                     # REQUIRED: create_fastapi_app(...) wiring + /health
+    environment.py             # REQUIRED: SecOpsEvidenceGymEnvironment(reset/step/state)
+    graders.py                 # REQUIRED: grade_easy/medium/hard + safe_reward clamp
+    tasks.py                   # OPTIONAL (high-leverage): scenario registry + seed sampling
+    safety.py                  # OPTIONAL (high-leverage): tool allowlist + sanitisation helpers
+    requirements.txt           # OPTIONAL (if Docker build uses it)
+    Dockerfile                 # REQUIRED (practically): HF Spaces docker build
+
+  tests/
+    test_api_contract.py       # smoke: reset/step/state doesn’t crash; reward range
+    test_graders.py            # unit: deterministic scoring + strict (0,1) clamp
+    test_seed_determinism.py   # unit: same seed → same evidence IDs
+```
+
+[PROPOSAL] Mandatory for hackathon success: `openenv.yaml`, server app wiring, three tasks+graders, Docker build success, `inference.py` with strict logs, and a README that makes the environment’s value obvious in <60 seconds.
+
+## Reward, grading, and anti-hallucination design
+
+**SECTION 10 — Reward Design**
+
+[SOURCED] OpenEnv leaves reward semantics to the environment; you are responsible for correctness scoring and determinism. citeturn18view0turn19search1  
+
+[SOURCED] Hackathon validation has shown strict “score must be between 0 and 1 (not 0.0 and not 1.0)” behaviour, and teams clamp rewards (e.g., 0.01–0.99). citeturn27search0turn26view0  
+
+[SOURCED] Empirical RL research in other domains (e.g., autonomous racing) shows reward design choices materially affect performance and generalisation, supporting the need for careful shaping rather than a single sparse terminal reward. citeturn15view2  
+
+[PROPOSAL] **Core principle:** correctness is **verifier-gated**, not language-judged. You can optionally add *format/style* checks, but never allow style to dominate correctness reward.
+
+### Reward structure (practical V1)
+
+[PROPOSAL] Normalise the final *task score* into `(0,1)` and keep per-step rewards small enough that summed episode reward stays in `(0,1)` as well (or only final reward is used, depending on your environment semantics). Use a single “score” to satisfy the validator and expose detailed breakdowns in `observation.metadata`.
+
+**Terminal (sparse) components** ✅  
+[PROPOSAL]  
+- `+0.60` if at least one ground-truth finding is verified and correctly described (type + impact).  
+- `+0.15` if the report includes **≥1 valid evidence ID** per finding and those IDs correspond to the right artefacts.  
+- `+0.15` if remediation is actionable (specific control, config, test).  
+- `-0.40` per hallucinated/unverified finding claimed in the report.  
+- `-0.20` if the agent fails to run `validate_finding()` before `submit_report()`.
+
+**Intermediate (dense) components** 🧭  
+[PROPOSAL]  
+- `+0.02` for discovering a *new* relevant evidence ID (first time only).  
+- `+0.03` for creating a well-formed candidate finding that references evidence IDs.  
+- `-0.01` per step (efficiency pressure).  
+- `-0.03` for repeating the same tool call (exact same args) beyond 2 times.  
+
+**False-positive penalties / anti-hallucination** 🧯  
+[PROPOSAL] A “hallucination” is operationally defined as: the report asserts a finding that is not in the environment’s `verified_findings` list. This is easy to compute deterministically and maps directly to your stated goal (“avoid hallucinating findings”).
+
+### Avoiding reward hacking
+
+[PROPOSAL] Hardening rules:  
+- Cap rewards from verbosity: extra words do not add points.  
+- Make evidence IDs required for high scores (prevents purely rhetorical “security speak”).  
+- Penalise calling `validate_finding()` repeatedly without new evidence.  
+- Reject “kitchen sink” reporting by penalising extra unverified findings.
+
+### Binary vs shaped reward
+
+[PROPOSAL] **Binary-only** (0/1) will be easy to implement but brittle for multi-step tool use; the agent gets no gradient for *how* to investigate efficiently.  
+
+[PROPOSAL] **Lightly shaped** (recommended) keeps correctness deterministic while providing enough signal to train investigation workflow (evidence collection, validation order, loop avoidance). This mirrors the broader lesson from reward engineering research: shaping and tuning can significantly alter learning outcomes. citeturn15view2  
+
+### Deterministic judge vs hybrid judge
+
+[PROPOSAL]  
+- **Strict deterministic judge (recommended V1):** all correctness via verifiers + string/structure checks.  
+- **Hybrid (stretch):** add a small LLM-based style score (e.g., clarity), heavily downweighted (≤0.05 of total) and never affecting pass/fail correctness.
+
+## Baseline inference pipeline and strict stdout logging
+
+**SECTION 11 — Baseline Inference Pipeline**
+
+[SOURCED] Hackathon requirements include: a reproducible `inference.py`, the OpenAI client requirement for LLM calls (using provided env vars), and strict stdout logging. citeturn3view6  
+
+[SOURCED] A concrete, hackathon-aligned stdout format has been used by validated submissions (example):  
+- `[START] task=<name> env=<benchmark> model=<model_name>`  
+- `[STEP] step=<n> action=<str> reward=<0.00> done=<true|false> error=<msg|null>`  
+- `[END] task=<name> success=<true|false> steps=<n> score=<0.00> rewards=<r1,r2,...>` citeturn22view1turn22view2  
+
+[SOURCED] The same example inference uses the OpenAI SDK, reading `API_BASE_URL`, `MODEL_NAME`, and `HF_TOKEN`. citeturn22view1turn22view4  
+
+### Responsibilities of `inference.py`
+
+[PROPOSAL] `inference.py` should:  
+- read env vars: `HF_TOKEN`, `API_BASE_URL`, `MODEL_NAME`, `ENV_URL` (and optionally `TASK_NAME` override),  
+- connect to the env via `.sync()` client,  
+- run tasks in a fixed order (easy → medium → hard),  
+- execute a bounded number of steps per task,  
+- log exactly one `[START]...` per task, one `[END]...` per task, and a `[STEP]...` per environment step,  
+- always exit with code 0 (even on failures) and log errors in the `[STEP] error=` field to avoid hard crashes.
+
+### Control flow (V1 baseline strategy)
+
+[PROPOSAL] Use a **hybrid baseline** that is reliable under time constraints:  
+- scripted tool sequence per task (fast, deterministic),  
+- one LLM call (optional) to draft the final report from gathered evidence (so the demo shows “agentic reasoning”),  
+- temperature fixed to 0 for reproducibility (and lower variance).  
+
+[SOURCED] Deterministic inference settings like `TEMPERATURE=0.0` are used in competitive OpenEnv hackathon baselines. citeturn20view0turn22view4  
+
+### Minimum viable baseline (must ship)
+
+[PROPOSAL] For each task:  
+1) `reset(task_id=<tier>)`  
+2) run 2–4 tool calls that are always relevant (e.g., `check_security_headers`, `search_repo`, etc.)  
+3) `create_finding(...)` using evidence IDs  
+4) `validate_finding(finding_id)`  
+5) `submit_report(report_json)`  
+
+### Stronger baseline (only if time permits)
+
+[PROPOSAL] Add one planning LLM call that chooses among tools based on the alert type, but still keep a hard step limit, and always include verifier validation before reporting.
+
+## Complete build, validation, deployment, and submission pipeline
+
+**SECTION 5 — Complete End-to-End Pipeline**
+
+[SOURCED] This pipeline is built to satisfy both OpenEnv conventions (init/push, typed models, FastAPI server) and hackathon validation constraints (tasks/graders, inference logging, runtime budgets). citeturn18view0turn19search2turn3view6turn22view1  
+
+### Phase goals, deliverables, verification (execution-ready)
+
+[PROPOSAL] The table below is the “do-this-in-order” execution plan. It is intentionally validator-first.
+
+| Phase | Goal | Deliverables | Files touched | Acceptance criteria | Main risks | How to verify |
+|---|---|---|---|---|---|---|
+| Scope lock | Freeze V1 to 3 tasks + bounded tools | 1-page spec + non-goals | README.md | No pentest/exploit scope; 3 tasks defined | Scope creep | Manual checklist |
+| Scaffold | Generate OpenEnv skeleton | Working importable package | openenv.yaml, models.py, client.py, server/* | `python -c "import ..."` succeeds | Wrong template/paths | Local import smoke test |
+| Environment core | Implement reset/step/state; tool router | Simulator runs end-to-end | server/environment.py | reset+step returns typed observation; no crashes | Action validation crashes | manual `curl` + python client |
+| Tasks + graders | Implement 3 graders + strict (0,1) clamp | `grade_easy/medium/hard` | server/graders.py, openenv.yaml | tasks discoverable; scores strictly in (0,1) | Validator fail-fast | unit tests + manual checks |
+| Baseline inference | Make inference reproducible + strict logs | inference.py | inference.py | prints correct `[START]/[STEP]/[END]` | log-parser failure | run script locally |
+| Local validation | Run OpenEnv build & validate | passes `openenv validate` | Dockerfile, server/app.py | validate passes locally | port mismatch | `openenv validate --url ...` |
+| Docker + HF | Deploy to Spaces | live endpoint | openenv push output | `/health` 200; reset+step works remotely | HF port/env mismatch | curl + python client |
+| Submission | Final narrative + demo | polished README + screenshots | README.md | demo works in <2 min | unclear story | run “demo script” |
+
+### Concrete build plan with commands
+
+[SOURCED] OpenEnv supports `openenv init` and `openenv push` and documents this as the standard creator workflow. citeturn18view0turn17view0  
+[SOURCED] The OpenEnv course also provides a grounded dev loop: `uv sync`, `uv run server`, `curl /health`, and Docker build/run commands. citeturn17view0  
+
+[PROPOSAL] Commands (copy/paste order):
+
+1) **Scaffold**
+```bash
+pip install openenv-core
+openenv init secops_evidence_gym
+cd secops_evidence_gym
+```
+[SOURCED] `openenv init` is the documented way to scaffold a new environment. citeturn18view0turn18view2  
+
+2) **Local dev install + run**
+```bash
+uv sync
+uv run server
+curl http://localhost:8000/health
+```
+[SOURCED] `uv run server` and `/health` checks are part of the recommended iteration loop in OpenEnv course materials. citeturn17view0  
+
+3) **Implement core files (edit)**
+- `models.py`: define `Action/Observation/State` dataclasses  
+- `server/environment.py`: implement reset/step/state + tool routing  
+- `server/graders.py`: implement `grade_easy/grade_medium/grade_hard` + `safe_reward()`  
+- `openenv.yaml`: add `tasks:` with grader import paths  
+
+[SOURCED] OpenEnv’s environment-building guide explicitly directs you to define models and implement `reset/step/state`, then wire a FastAPI app. citeturn19search1  
+[SOURCED] A validator-aligned `openenv.yaml` with `spec_version`, `runtime`, `app`, `port`, and `tasks` exists in deep-validation passing examples. citeturn23view0  
+
+4) **Build + validate (local)**
+```bash
+openenv build
+openenv validate --verbose
+```
+[SOURCED] `openenv build` and `openenv validate` are part of OpenEnv’s recommended validation workflow. citeturn19search2  
+
+5) **Docker build/run smoke test**
+```bash
+docker build -t secops-evidence-gym:latest -f server/Dockerfile .
+docker run -p 8000:8000 secops-evidence-gym:latest
+curl http://localhost:8000/health
+```
+[SOURCED] This `docker build -f server/Dockerfile .` pattern is directly shown in OpenEnv deployment course material. citeturn17view0  
+
+6) **Run inference locally**
+```bash
+export HF_TOKEN="..."
+export API_BASE_URL="..."
+export MODEL_NAME="..."
+export ENV_URL="http://localhost:8000"
+python inference.py
+```
+[SOURCED] These env var names and OpenAI SDK usage are consistent with hackathon guidance and existing inference implementations. citeturn3view6turn22view4  
+
+7) **Deploy to Hugging Face Spaces**
+```bash
+openenv push --repo-id <your-hf-username>/secops-evidence-gym
+```
+[SOURCED] `openenv push` is described as the fastest path to deploy to **Hugging Face Spaces**. citeturn17view0turn18view0  
+
+### Testing and validation plan (high-signal)
+
+[SOURCED] OpenEnv stresses predictable API behaviour and type-safe contracts; hackathon validation is fail-fast. citeturn18view0turn27search0  
+
+[PROPOSAL] Test layers (in priority order):  
+- **API contract smoke tests:** reset/step/state return valid JSON; never crash on invalid tool name (should return an observation with an error field).  
+- **Grader tests:** for each task, verify (a) correctness cases score high, (b) hallucination cases score low, (c) score always ∈ (0,1).  
+- **Seed determinism tests:** same `seed` produces same evidence IDs and same verifier outputs.  
+- **Runtime test:** run `inference.py` end-to-end and assert wall-clock < 2 minutes locally; assume < 20 minutes on grader infra even with cold starts. citeturn3view6turn22view4  
+- **Reward sanity tests:** ensure reward increases monotonically with verified correctness; fails if verbosity alone increases reward.
+
+## Submission packaging, execution roadmap, real-world usefulness, and failure modes
+
+**SECTION 14 — README / Demo / Submission Narrative**  
+[SOURCED] Judges likely assess both the environment’s technical correctness (programmatic checks) and qualitative merit (LLM scoring / narrative). citeturn3view7  
+
+[PROPOSAL] README structure that “feels like a winner” 🏆:  
+- **Hero block:** one-paragraph pitch + why it’s real-world + safety claim.  
+- **Two-minute demo:** copy/paste commands + expected output snippet with `[START]/[STEP]/[END]`.  
+- **Environment contract:** action schema, observation schema, task list.  
+- **Grading:** explain deterministic verifiers + hallucination penalties.  
+- **Safety & isolation:** explicit exclusions (no egress, no shell, synthetic artefacts).  
+- **Real-world relevance:** how this benchmarks/reporting maps to security workflows (triage, evidence, remediation).  
+- **Screenshots:** web UI (optional) + an evidence trace + one scored report example.  
+
+**SECTION 15 — Project Management Plan**  
+[PROPOSAL] Day-by-day (assuming a hackathon-style sprint):
+
+- **Day 0 (scope lock + scaffold):** environment skeleton, `openenv.yaml` with 3 tasks, stub graders returning 0.5 (clamped), server runs locally.  
+- **Day 1 (determinism + validator):** implement scenario generator, evidence registry, verifiers, and strict (0,1) scoring; pass `openenv validate`.  
+- **Day 2 (baseline + polish):** implement `inference.py` strict logs; deploy to Spaces; polish README + demo artefacts.
+
+[PROPOSAL] Critical path: `openenv.yaml tasks+graders` → grader clamp `(0,1)` → inference stdout format → Docker+Spaces deployment. (Everything else is secondary.)
+
+**SECTION 16 — Real-World Usefulness Plan**  
+[SOURCED] NIST’s testing guide emphasises planning, conducting tests, analysing findings, and developing mitigation strategies; your environment’s “evidence → remediation” focus aligns with that lifecycle without requiring offensive exploitation. citeturn29search8turn29search0  
+
+[PROPOSAL] Who would care after the hackathon:  
+- security engineering teams evaluating agentic “triage + reporting” reliability,  
+- LLM tooling teams wanting benchmarks for **non-hallucinating, evidence-grounded** outputs,  
+- training teams building safe cyber ranges (without weaponisation).
+
+[PROPOSAL] Post-hackathon upgrades (highest leverage):  
+- export trajectories as JSONL for offline training,  
+- add more scenario families (still safe) and a held-out split for generalisation,  
+- integrate with RL trainers (e.g., TRL’s OpenEnv integration) to show real training curves. citeturn19search6turn10view0  
+
+[SOURCED] PenGym provides evidence that realism/faithfulness of environments can affect transfer and stability when moving from simulation to more realistic settings—so you should roadmap a “higher fidelity mode” (still safe) later, not in V1. citeturn15view0  
+
+**SECTION 17 — Why the naive version would fail**  
+[PROPOSAL] Top failure patterns (and why they kill submissions):  
+- Too broad (full cyber range, live services): fails time/infra constraints. citeturn3view6turn10view0  
+- Fuzzy grading (LLM-only judging): non-deterministic, easy to game.  
+- Unbounded tools (shell/network): unsafe + untrainable action space.  
+- Scores at exactly 0.0 or 1.0: fail-fast “out of range” validator. citeturn27search0turn26view0  
+- Inference logs not parseable: phase-1 failure even if env is good. citeturn3view6turn22view1  
+- Port / health issues on Spaces: container “works locally” but fails remotely. citeturn17view0turn20view0  
+
+**SECTION 18 — Final Recommendation**
+
+[PROPOSAL] **What should you build?**  
+Build **SecOps Evidence Gym**: a deterministic, safe, sandbox-only cyber analyst environment focused on evidence collection, verifier validation, and remediation reporting.
+
+[PROPOSAL] **What should V1 include? (minimum winning set)**  
+- OpenEnv-compliant FastAPI env with typed models and `reset/step/state`. citeturn18view0turn19search1  
+- `openenv.yaml` with **3 tasks + graders**. citeturn23view0turn3view6  
+- Deterministic verifiers + strict score clamp to `(0,1)`. citeturn27search0turn26view0  
+- Baseline `inference.py` with strict `[START]/[STEP]/[END]` logging + OpenAI SDK usage for any LLM calls. citeturn3view6turn22view1turn22view4  
+- HF Spaces deployment with a working `/health`. citeturn17view0turn20view0  
+
+[PROPOSAL] **What should you cut?**  
+- Any real pentesting/offensive content, any arbitrary command execution, any live targets, any correctness scoring via an LLM judge.
+
+[PROPOSAL] **Top 5 implementation decisions that matter most**  
+1) Validator-safe `openenv.yaml` tasks+graders wiring. citeturn23view0  
+2) Score/range compliance: clamp to `(0,1)` everywhere. citeturn27search0turn26view0  
+3) Strict stdout format in `inference.py`. citeturn22view1turn22view2  
+4) Deterministic verifiers as the source of truth.  
+5) Bounded tool set (≤8 tools) with anti-loop penalties.
+
+[PROPOSAL] **Minimum viable winning submission**  
+A V1 with 3 tasks, deterministic graders, bounded tools, strict inference logging, and a polished README + demo trace.
+
+[PROPOSAL] **Minimum viable real-world useful submission**  
+The same V1, plus: seed determinism, trajectory export, and a clear “how to add new scenarios” contributor guide.
+
+[PROPOSAL] **If you only have time for 20% of ambition—do this exact 20%:**  
+- Implement **one** robust multi-step loop (tools → validate → report)  
+- Implement **exactly 3** tasks (easy/medium/hard)  
+- Make graders deterministic and validator-safe  
+- Make deployment + inference bulletproof  
+Everything else is stretch.
+
+**Confidence (my estimate): 8.4/10** ✅🔥
+
+## Sources and credibility ratings (with exact links)
+
+[SOURCED] Ratings are my judgement of authority + relevance for this hackathon context (0–10). URLs are provided verbatim in code form.
+
+### Tier 1 (official OpenEnv + hackathon dashboard)
+- Credibility **9.5/10** — `https://github.com/meta-pytorch/OpenEnv` citeturn18view0  
+- Credibility **9.0/10** — `https://github.com/meta-pytorch/OpenEnv/blob/main/envs/README.md` citeturn19search1  
+- Credibility **8.5/10** — `https://github.com/meta-pytorch/OpenEnv/blob/main/.claude/skills/generate-openenv-env/SKILL.md` citeturn19search2  
+- Credibility **9.0/10** — `https://www.scaler.com/school-of-technology/meta-pytorch-hackathon/dashboard` citeturn1view0turn3view6turn3view7  
+
+### Tier 2 (strong community exemplars)
+- Credibility **8.5/10** — `https://github.com/sid-rp/kube-sre-gym` citeturn10view0  
+- Credibility **8.0/10** — `https://huggingface.co/openenv-community` citeturn14view0  
+- Credibility **7.5/10** — `https://github.com/Harikishanth/Incident-Triage-Environment` citeturn20view0turn23view0turn22view1  
+
+### Tier 3 (peer-reviewed / primary references for design constraints)
+- Credibility **8.5/10** — PenGym (Computers & Security, open access): `https://www.sciencedirect.com/science/article/pii/S0167404824004450` citeturn15view0  
+- Credibility **8.0/10** — Reward design + generalisation (Scientific Reports, 2025): `https://www.nature.com/articles/s41598-025-27702-6` citeturn15view2  
+- Credibility **8.5/10** — AMaze (JOSS, 2025): `https://joss.theoj.org/papers/10.21105/joss.07208` citeturn16search7  
+- Credibility **9.5/10** — NIST SP 800-115: `https://csrc.nist.gov/pubs/sp/800/115/final` citeturn29search8  
+- Credibility **9.0/10** — NIST “Cyber Range: A Guide” (PDF landing): `https://www.nist.gov/document/cyber-range` citeturn29search1  
+- Credibility **7.5/10** — “Cybersecurity of Cyber Ranges: Threats and Mitigations” (IJISR, 2022 PDF): `https://infonomics-society.org/wp-content/uploads/Cybersecurity-of-Cyber-Ranges.pdf` citeturn29search2  
+
+### Tier 4 (useful validator “ground truth” signals from the field)
+- Credibility **6.5/10** — Validator failure mode discussion (score must be strictly between 0 and 1): `https://www.reddit.com/r/pytorch/comments/1shi767/meta_x_pytorch_x_sst_x_openenv_hackathon_phase_2/` citeturn27search0  
+- Credibility **7.0/10** — Strict logging format reference via a verified submission’s `inference.py`: `https://github.com/Harikishanth/Incident-Triage-Environment/blob/main/inference.py` citeturn22view1turn22view2  
+
+### Uploaded reference you provided
 - Credibility **7.0/10** (useful as a design draft; not independently authoritative) — `deep-research-report (2).md` fileciteturn2file0

inference.py

@@ -1,290 +1,290 @@
-#!/usr/bin/env python3
-"""Model-backed baseline inference for the Cyber Analyst OpenEnv environment."""
-
-from __future__ import annotations
-
-import json
-import os
-import sys
-import textwrap
-from dataclasses import dataclass
-from pathlib import Path
-from typing import Any
-
-from openai import OpenAI
-
-PACKAGE_PARENT = Path(__file__).resolve().parent.parent
-if str(PACKAGE_PARENT) not in sys.path:
-    sys.path.insert(0, str(PACKAGE_PARENT))
-
-from Cyber_analyst import CyberAnalystAction, CyberAnalystEnv, CyberAnalystObservation
-
-
-ENV_NAME = "Cyber_analyst"
-ENV_URL = os.getenv("ENV_URL", "http://localhost:8000")
-API_BASE_URL = os.getenv("API_BASE_URL", "https://router.huggingface.co/v1")
-MODEL_NAME = os.getenv("MODEL_NAME", "google/gemma-4-31B-it:fastest")
-TEMPERATURE = float(os.getenv("TEMPERATURE", "0.0"))
-MAX_TOKENS = int(os.getenv("MAX_TOKENS", "512"))
-MAX_STEPS = int(os.getenv("MAX_STEPS", "12"))
-SEED = int(os.getenv("SEED", "7"))
-TASK_IDS = [
-    "secret_exposure_easy",
-    "missing_security_headers_medium",
-    "authz_boundary_hard",
-]
-
-SYSTEM_PROMPT = textwrap.dedent(
-    """
-    You are running a bounded Cyber Analyst benchmark. You may only choose one
-    tool call from the provided tool catalog per turn. All evidence is synthetic;
-    do not request shell access, live network access, or external investigation.
-
-    Return exactly one compact JSON object and no surrounding text:
-    {"tool_name":"<tool name>","args":{...}}
-
-    To complete an episode, first discover relevant evidence, then create and
-    validate a finding, then submit a report_json with findings that include
-    finding_type, evidence_ids, impact, and remediation.
-    """
-).strip()
-
-
-@dataclass(frozen=True)
-class LLMConfig:
-    base_url: str
-    model_name: str
-    temperature: float
-    max_tokens: int
-
-
-class ModelActionError(RuntimeError):
-    """Raised when the model cannot provide a valid benchmark action."""
-
-
-def build_llm_config() -> LLMConfig:
-    return LLMConfig(
-        base_url=API_BASE_URL,
-        model_name=MODEL_NAME,
-        temperature=TEMPERATURE,
-        max_tokens=MAX_TOKENS,
-    )
-
-
-def build_openai_client() -> OpenAI:
-    """Return an OpenAI-compatible client for the Hugging Face Router."""
-
-    return OpenAI(base_url=API_BASE_URL, api_key=os.environ["HF_TOKEN"])
-
-
-def single_line(value: str) -> str:
-    return " ".join(str(value).split())
-
-
-def action_to_log(action: CyberAnalystAction) -> str:
-    payload = {"tool_name": action.tool_name, "args": action.args}
-    return single_line(json.dumps(payload, sort_keys=True, separators=(",", ":")))
-
-
-def log_start(task_id: str, llm_config: LLMConfig) -> None:
-    print(
-        f"[START] task={task_id} env={ENV_NAME} model={llm_config.model_name}",
-        flush=True,
-    )
-
-
-def log_step(
-    step: int, action: CyberAnalystAction, reward: float | None, done: bool, error: str
-) -> None:
-    reward_value = 0.0 if reward is None else float(reward)
-    error_value = single_line(error) if error else "null"
-    print(
-        f"[STEP] step={step} action={action_to_log(action)} "
-        f"reward={reward_value:.2f} done={str(done).lower()} error={error_value}",
-        flush=True,
-    )
-
-
-def log_end(task_id: str, success: bool, steps: int, score: float, rewards: list[float]) -> None:
-    rewards_text = ",".join(f"{reward:.2f}" for reward in rewards)
-    print(
-        f"[END] task={task_id} success={str(success).lower()} "
-        f"steps={steps} score={score:.2f} rewards={rewards_text}",
-        flush=True,
-    )
-
-
-def observation_payload(obs: CyberAnalystObservation) -> dict[str, Any]:
-    return {
-        "task_id": obs.task_id,
-        "alert": obs.alert,
-        "phase": obs.phase,
-        "tool_catalog": obs.tool_catalog,
-        "tool_result": obs.tool_result,
-        "evidence_ids": obs.evidence_ids,
-        "candidate_findings": obs.candidate_findings,
-        "verified_findings": obs.verified_findings,
-        "step_budget_remaining": obs.step_budget_remaining,
-        "score_breakdown": obs.score_breakdown,
-        "error": obs.error,
-    }
-
-
-def build_user_prompt(task_id: str, step: int, obs: CyberAnalystObservation) -> str:
-    payload = {
-        "task_id": task_id,
-        "step": step,
-        "observation": observation_payload(obs),
-    }
-    return textwrap.dedent(
-        f"""
-        Choose the next benchmark tool call.
-        Current state JSON:
-        {json.dumps(payload, sort_keys=True)}
-        """
-    ).strip()
-
-
-def extract_json_object(text: str) -> dict[str, Any]:
-    content = text.strip()
-    if content.startswith("```"):
-        lines = content.splitlines()
-        if lines and lines[0].startswith("```"):
-            lines = lines[1:]
-        if lines and lines[-1].startswith("```"):
-            lines = lines[:-1]
-        content = "\n".join(lines).strip()
-
-    try:
-        decoded = json.loads(content)
-    except json.JSONDecodeError as exc:
-        raise ModelActionError(f"model_parse_error: {exc.msg}") from exc
-
-    if not isinstance(decoded, dict):
-        raise ModelActionError("model_parse_error: response is not a JSON object")
-    return decoded
-
-
-def parse_model_action(text: str) -> CyberAnalystAction:
-    payload = extract_json_object(text)
-    tool_name = payload.get("tool_name")
-    args = payload.get("args", {})
-
-    if not isinstance(tool_name, str) or not tool_name:
-        raise ModelActionError("model_parse_error: missing tool_name")
-    if not isinstance(args, dict):
-        raise ModelActionError("model_parse_error: args must be an object")
-
-    return CyberAnalystAction(tool_name=tool_name, args=args)
-
-
-def get_model_action(
-    client: OpenAI,
-    llm_config: LLMConfig,
-    task_id: str,
-    step: int,
-    obs: CyberAnalystObservation,
-) -> CyberAnalystAction:
-    try:
-        completion = client.chat.completions.create(
-            model=llm_config.model_name,
-            messages=[
-                {"role": "system", "content": SYSTEM_PROMPT},
-                {"role": "user", "content": build_user_prompt(task_id, step, obs)},
-            ],
-            temperature=llm_config.temperature,
-            max_tokens=llm_config.max_tokens,
-            stream=False,
-        )
-    except Exception as exc:
-        raise ModelActionError(f"model_request_error: {exc}") from exc
-
-    text = (completion.choices[0].message.content or "").strip()
-    if not text:
-        raise ModelActionError("model_parse_error: empty response")
-    return parse_model_action(text)
-
-
-def error_action(error: Exception) -> CyberAnalystAction:
-    message = single_line(str(error))
-    if message.startswith("model_request_error"):
-        tool_name = "model_request_error"
-    elif message.startswith("model_parse_error"):
-        tool_name = "model_parse_error"
-    else:
-        tool_name = "model_action_error"
-    return CyberAnalystAction(
-        tool_name=tool_name,
-        args={"message": message[:500]},
-    )
-
-
-def run_task(task_id: str, client: OpenAI, llm_config: LLMConfig) -> None:
-    log_start(task_id, llm_config)
-    rewards: list[float] = []
-    steps_taken = 0
-    final_score = 0.01
-    success = False
-
-    try:
-        with CyberAnalystEnv(base_url=ENV_URL).sync() as env:
-            reset_result = env.reset(task_id=task_id, seed=SEED)
-            obs = reset_result.observation
-
-            for step in range(1, MAX_STEPS + 1):
-                if obs.done:
-                    break
-
-                model_failed = False
-                try:
-                    action = get_model_action(client, llm_config, task_id, step, obs)
-                except ModelActionError as exc:
-                    action = error_action(exc)
-                    model_failed = True
-
-                result = env.step(action)
-                obs = result.observation
-                reward = float(result.reward or 0.0)
-                rewards.append(reward)
-                steps_taken = step
-
-                log_step(step, action, result.reward, result.done, obs.error)
-
-                if isinstance(obs.tool_result, dict) and "score" in obs.tool_result:
-                    final_score = float(obs.tool_result["score"])
-
-                if result.done or model_failed:
-                    success = final_score > 0.5
-                    break
-
-    except Exception as exc:
-        action = CyberAnalystAction(
-            tool_name="runtime_error",
-            args={"message": single_line(str(exc))[:500]},
-        )
-        steps_taken = max(steps_taken, 1)
-        rewards.append(0.01)
-        log_step(steps_taken, action, 0.01, True, single_line(str(exc)))
-
-    log_end(task_id, success, steps_taken, final_score, rewards)
-
-
-def selected_task_ids() -> list[str]:
-    task_override = os.getenv("TASK_NAME")
-    return [task_override] if task_override else TASK_IDS
-
-
-def main() -> None:
-    llm_config = build_llm_config()
-    try:
-        client = build_openai_client()
-    except KeyError:
-        print("HF_TOKEN must be set for inference.", file=sys.stderr, flush=True)
-        raise SystemExit(2) from None
-
-    for task_id in selected_task_ids():
-        run_task(task_id, client, llm_config)
-
-
-if __name__ == "__main__":
-    main()
+#!/usr/bin/env python3
+"""Model-backed baseline inference for the Cyber Analyst OpenEnv environment."""
+
+from __future__ import annotations
+
+import json
+import os
+import sys
+import textwrap
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any
+
+from openai import OpenAI
+
+PACKAGE_PARENT = Path(__file__).resolve().parent.parent
+if str(PACKAGE_PARENT) not in sys.path:
+    sys.path.insert(0, str(PACKAGE_PARENT))
+
+from Cyber_analyst import CyberAnalystAction, CyberAnalystEnv, CyberAnalystObservation
+
+
+ENV_NAME = "Cyber_analyst"
+ENV_URL = os.getenv("ENV_URL", "http://localhost:8000")
+API_BASE_URL = os.getenv("API_BASE_URL", "https://router.huggingface.co/v1")
+MODEL_NAME = os.getenv("MODEL_NAME", "google/gemma-4-31B-it:fastest")
+TEMPERATURE = float(os.getenv("TEMPERATURE", "0.0"))
+MAX_TOKENS = int(os.getenv("MAX_TOKENS", "512"))
+MAX_STEPS = int(os.getenv("MAX_STEPS", "12"))
+SEED = int(os.getenv("SEED", "7"))
+TASK_IDS = [
+    "secret_exposure_easy",
+    "missing_security_headers_medium",
+    "authz_boundary_hard",
+]
+
+SYSTEM_PROMPT = textwrap.dedent(
+    """
+    You are running a bounded Cyber Analyst benchmark. You may only choose one
+    tool call from the provided tool catalog per turn. All evidence is synthetic;
+    do not request shell access, live network access, or external investigation.
+
+    Return exactly one compact JSON object and no surrounding text:
+    {"tool_name":"<tool name>","args":{...}}
+
+    To complete an episode, first discover relevant evidence, then create and
+    validate a finding, then submit a report_json with findings that include
+    finding_type, evidence_ids, impact, and remediation.
+    """
+).strip()
+
+
+@dataclass(frozen=True)
+class LLMConfig:
+    base_url: str
+    model_name: str
+    temperature: float
+    max_tokens: int
+
+
+class ModelActionError(RuntimeError):
+    """Raised when the model cannot provide a valid benchmark action."""
+
+
+def build_llm_config() -> LLMConfig:
+    return LLMConfig(
+        base_url=API_BASE_URL,
+        model_name=MODEL_NAME,
+        temperature=TEMPERATURE,
+        max_tokens=MAX_TOKENS,
+    )
+
+
+def build_openai_client() -> OpenAI:
+    """Return an OpenAI-compatible client for the Hugging Face Router."""
+
+    return OpenAI(base_url=API_BASE_URL, api_key=os.environ["HF_TOKEN"])
+
+
+def single_line(value: str) -> str:
+    return " ".join(str(value).split())
+
+
+def action_to_log(action: CyberAnalystAction) -> str:
+    payload = {"tool_name": action.tool_name, "args": action.args}
+    return single_line(json.dumps(payload, sort_keys=True, separators=(",", ":")))
+
+
+def log_start(task_id: str, llm_config: LLMConfig) -> None:
+    print(
+        f"[START] task={task_id} env={ENV_NAME} model={llm_config.model_name}",
+        flush=True,
+    )
+
+
+def log_step(
+    step: int, action: CyberAnalystAction, reward: float | None, done: bool, error: str
+) -> None:
+    reward_value = 0.0 if reward is None else float(reward)
+    error_value = single_line(error) if error else "null"
+    print(
+        f"[STEP] step={step} action={action_to_log(action)} "
+        f"reward={reward_value:.2f} done={str(done).lower()} error={error_value}",
+        flush=True,
+    )
+
+
+def log_end(task_id: str, success: bool, steps: int, score: float, rewards: list[float]) -> None:
+    rewards_text = ",".join(f"{reward:.2f}" for reward in rewards)
+    print(
+        f"[END] task={task_id} success={str(success).lower()} "
+        f"steps={steps} score={score:.2f} rewards={rewards_text}",
+        flush=True,
+    )
+
+
+def observation_payload(obs: CyberAnalystObservation) -> dict[str, Any]:
+    return {
+        "task_id": obs.task_id,
+        "alert": obs.alert,
+        "phase": obs.phase,
+        "tool_catalog": obs.tool_catalog,
+        "tool_result": obs.tool_result,
+        "evidence_ids": obs.evidence_ids,
+        "candidate_findings": obs.candidate_findings,
+        "verified_findings": obs.verified_findings,
+        "step_budget_remaining": obs.step_budget_remaining,
+        "score_breakdown": obs.score_breakdown,
+        "error": obs.error,
+    }
+
+
+def build_user_prompt(task_id: str, step: int, obs: CyberAnalystObservation) -> str:
+    payload = {
+        "task_id": task_id,
+        "step": step,
+        "observation": observation_payload(obs),
+    }
+    return textwrap.dedent(
+        f"""
+        Choose the next benchmark tool call.
+        Current state JSON:
+        {json.dumps(payload, sort_keys=True)}
+        """
+    ).strip()
+
+
+def extract_json_object(text: str) -> dict[str, Any]:
+    content = text.strip()
+    if content.startswith("```"):
+        lines = content.splitlines()
+        if lines and lines[0].startswith("```"):
+            lines = lines[1:]
+        if lines and lines[-1].startswith("```"):
+            lines = lines[:-1]
+        content = "\n".join(lines).strip()
+
+    try:
+        decoded = json.loads(content)
+    except json.JSONDecodeError as exc:
+        raise ModelActionError(f"model_parse_error: {exc.msg}") from exc
+
+    if not isinstance(decoded, dict):
+        raise ModelActionError("model_parse_error: response is not a JSON object")
+    return decoded
+
+
+def parse_model_action(text: str) -> CyberAnalystAction:
+    payload = extract_json_object(text)
+    tool_name = payload.get("tool_name")
+    args = payload.get("args", {})
+
+    if not isinstance(tool_name, str) or not tool_name:
+        raise ModelActionError("model_parse_error: missing tool_name")
+    if not isinstance(args, dict):
+        raise ModelActionError("model_parse_error: args must be an object")
+
+    return CyberAnalystAction(tool_name=tool_name, args=args)
+
+
+def get_model_action(
+    client: OpenAI,
+    llm_config: LLMConfig,
+    task_id: str,
+    step: int,
+    obs: CyberAnalystObservation,
+) -> CyberAnalystAction:
+    try:
+        completion = client.chat.completions.create(
+            model=llm_config.model_name,
+            messages=[
+                {"role": "system", "content": SYSTEM_PROMPT},
+                {"role": "user", "content": build_user_prompt(task_id, step, obs)},
+            ],
+            temperature=llm_config.temperature,
+            max_tokens=llm_config.max_tokens,
+            stream=False,
+        )
+    except Exception as exc:
+        raise ModelActionError(f"model_request_error: {exc}") from exc
+
+    text = (completion.choices[0].message.content or "").strip()
+    if not text:
+        raise ModelActionError("model_parse_error: empty response")
+    return parse_model_action(text)
+
+
+def error_action(error: Exception) -> CyberAnalystAction:
+    message = single_line(str(error))
+    if message.startswith("model_request_error"):
+        tool_name = "model_request_error"
+    elif message.startswith("model_parse_error"):
+        tool_name = "model_parse_error"
+    else:
+        tool_name = "model_action_error"
+    return CyberAnalystAction(
+        tool_name=tool_name,
+        args={"message": message[:500]},
+    )
+
+
+def run_task(task_id: str, client: OpenAI, llm_config: LLMConfig) -> None:
+    log_start(task_id, llm_config)
+    rewards: list[float] = []
+    steps_taken = 0
+    final_score = 0.01
+    success = False
+
+    try:
+        with CyberAnalystEnv(base_url=ENV_URL).sync() as env:
+            reset_result = env.reset(task_id=task_id, seed=SEED)
+            obs = reset_result.observation
+
+            for step in range(1, MAX_STEPS + 1):
+                if obs.done:
+                    break
+
+                model_failed = False
+                try:
+                    action = get_model_action(client, llm_config, task_id, step, obs)
+                except ModelActionError as exc:
+                    action = error_action(exc)
+                    model_failed = True
+
+                result = env.step(action)
+                obs = result.observation
+                reward = float(result.reward or 0.0)
+                rewards.append(reward)
+                steps_taken = step
+
+                log_step(step, action, result.reward, result.done, obs.error)
+
+                if isinstance(obs.tool_result, dict) and "score" in obs.tool_result:
+                    final_score = float(obs.tool_result["score"])
+
+                if result.done or model_failed:
+                    success = final_score > 0.5
+                    break
+
+    except Exception as exc:
+        action = CyberAnalystAction(
+            tool_name="runtime_error",
+            args={"message": single_line(str(exc))[:500]},
+        )
+        steps_taken = max(steps_taken, 1)
+        rewards.append(0.01)
+        log_step(steps_taken, action, 0.01, True, single_line(str(exc)))
+
+    log_end(task_id, success, steps_taken, final_score, rewards)
+
+
+def selected_task_ids() -> list[str]:
+    task_override = os.getenv("TASK_NAME")
+    return [task_override] if task_override else TASK_IDS
+
+
+def main() -> None:
+    llm_config = build_llm_config()
+    try:
+        client = build_openai_client()
+    except KeyError:
+        print("HF_TOKEN must be set for inference.", file=sys.stderr, flush=True)
+        raise SystemExit(2) from None
+
+    for task_id in selected_task_ids():
+        run_task(task_id, client, llm_config)
+
+
+if __name__ == "__main__":
+    main()

models.py

@@ -1,64 +1,64 @@
-# Copyright (c) Meta Platforms, Inc. and affiliates.
-# All rights reserved.
-#
-# This source code is licensed under the BSD-style license found in the
-# LICENSE file in the root directory of this source tree.
-
-"""Typed models for the Cyber Analyst OpenEnv environment."""
-
-from typing import Any
-
-from openenv.core.env_server.types import Action, Observation, State
-from pydantic import Field
-
-
-class CyberAnalystAction(Action):
-    """A bounded simulator tool call."""
-
-    tool_name: str = Field(..., description="Name of the approved simulator tool")
-    args: dict[str, Any] = Field(
-        default_factory=dict,
-        description="Tool arguments. The environment ignores unsupported keys.",
-    )
-
-
-class CyberAnalystObservation(Observation):
-    """Observation returned after reset or an environment step."""
-
-    task_id: str = Field(default="", description="Current benchmark task id")
-    alert: str = Field(default="", description="Initial alert or task prompt")
-    phase: str = Field(default="investigate", description="Current episode phase")
-    tool_catalog: list[dict[str, Any]] = Field(
-        default_factory=list, description="Approved tools and their schemas"
-    )
-    tool_result: dict[str, Any] = Field(
-        default_factory=dict, description="Result returned by the latest tool call"
-    )
-    evidence_ids: list[str] = Field(
-        default_factory=list, description="Evidence ids discovered so far"
-    )
-    verified_findings: list[dict[str, Any]] = Field(
-        default_factory=list, description="Verifier-confirmed findings"
-    )
-    candidate_findings: list[dict[str, Any]] = Field(
-        default_factory=list, description="Candidate findings created by the agent"
-    )
-    step_budget_remaining: int = Field(
-        default=0, ge=0, description="Steps remaining before timeout"
-    )
-    score_breakdown: dict[str, Any] = Field(
-        default_factory=dict, description="Deterministic reward/score explanation"
-    )
-    error: str = Field(default="", description="Non-fatal environment error, if any")
-
-
-class CyberAnalystState(State):
-    """State summary exposed via the OpenEnv state endpoint."""
-
-    task_id: str = Field(default="", description="Current benchmark task id")
-    seed: int | None = Field(default=None, description="Current deterministic seed")
-    phase: str = Field(default="investigate", description="Current episode phase")
-    step_budget_remaining: int = Field(default=0, ge=0)
-    recent_evidence_ids: list[str] = Field(default_factory=list)
-    verified_finding_ids: list[str] = Field(default_factory=list)
-    done: bool = Field(default=False)
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+"""Typed models for the Cyber Analyst OpenEnv environment."""
+
+from typing import Any
+
+from openenv.core.env_server.types import Action, Observation, State
+from pydantic import Field
+
+
+class CyberAnalystAction(Action):
+    """A bounded simulator tool call."""
+
+    tool_name: str = Field(..., description="Name of the approved simulator tool")
+    args: dict[str, Any] = Field(
+        default_factory=dict,
+        description="Tool arguments. The environment ignores unsupported keys.",
+    )
+
+
+class CyberAnalystObservation(Observation):
+    """Observation returned after reset or an environment step."""
+
+    task_id: str = Field(default="", description="Current benchmark task id")
+    alert: str = Field(default="", description="Initial alert or task prompt")
+    phase: str = Field(default="investigate", description="Current episode phase")
+    tool_catalog: list[dict[str, Any]] = Field(
+        default_factory=list, description="Approved tools and their schemas"
+    )
+    tool_result: dict[str, Any] = Field(
+        default_factory=dict, description="Result returned by the latest tool call"
+    )
+    evidence_ids: list[str] = Field(
+        default_factory=list, description="Evidence ids discovered so far"
+    )
+    verified_findings: list[dict[str, Any]] = Field(
+        default_factory=list, description="Verifier-confirmed findings"
+    )
+    candidate_findings: list[dict[str, Any]] = Field(
+        default_factory=list, description="Candidate findings created by the agent"
+    )
+    step_budget_remaining: int = Field(
+        default=0, ge=0, description="Steps remaining before timeout"
+    )
+    score_breakdown: dict[str, Any] = Field(
+        default_factory=dict, description="Deterministic reward/score explanation"
+    )
+    error: str = Field(default="", description="Non-fatal environment error, if any")
+
+
+class CyberAnalystState(State):
+    """State summary exposed via the OpenEnv state endpoint."""
+
+    task_id: str = Field(default="", description="Current benchmark task id")
+    seed: int | None = Field(default=None, description="Current deterministic seed")
+    phase: str = Field(default="investigate", description="Current episode phase")
+    step_budget_remaining: int = Field(default=0, ge=0)
+    recent_evidence_ids: list[str] = Field(default_factory=list)
+    verified_finding_ids: list[str] = Field(default_factory=list)
+    done: bool = Field(default=False)

openenv.yaml

@@ -1,16 +1,16 @@
-spec_version: 1
-name: Cyber_analyst
-type: space
-runtime: fastapi
-app: server.app:app
-port: 8000
-tasks:
-  - id: secret_exposure_easy
-    description: Detect a leaked synthetic API key in a repo snapshot and submit rotation/removal remediation.
-    grader: server.graders:grade_secret_exposure_easy
-  - id: missing_security_headers_medium
-    description: Detect missing HSTS/CSP headers in a synthetic gateway header snapshot and submit remediation.
-    grader: server.graders:grade_missing_security_headers_medium
-  - id: authz_boundary_hard
-    description: Detect an admin route role-policy mismatch and submit least-privilege remediation.
-    grader: server.graders:grade_authz_boundary_hard
+spec_version: 1
+name: Cyber_analyst
+type: space
+runtime: fastapi
+app: server.app:app
+port: 8000
+tasks:
+  - id: secret_exposure_easy
+    description: Detect a leaked synthetic API key in a repo snapshot and submit rotation/removal remediation.
+    grader: server.graders:grade_secret_exposure_easy
+  - id: missing_security_headers_medium
+    description: Detect missing HSTS/CSP headers in a synthetic gateway header snapshot and submit remediation.
+    grader: server.graders:grade_missing_security_headers_medium
+  - id: authz_boundary_hard
+    description: Detect an admin route role-policy mismatch and submit least-privilege remediation.
+    grader: server.graders:grade_authz_boundary_hard

pyproject.toml

@@ -1,39 +1,39 @@
-# Copyright (c) Meta Platforms, Inc. and affiliates.
-# All rights reserved.
-#
-# This source code is licensed under the BSD-style license found in the
-# LICENSE file in the root directory of this source tree.
-
-[build-system]
-requires = ["setuptools>=45", "wheel"]
-build-backend = "setuptools.build_meta"
-
-[project]
-name = "openenv-Cyber_analyst"
-version = "0.1.0"
-description = "Cyber Analyst environment for OpenEnv"
-requires-python = ">=3.10"
-dependencies = [
-    # Core OpenEnv runtime (provides FastAPI server + HTTP client types)
-    # install from github
-    # "openenv-core[core] @ git+https://github.com/meta-pytorch/OpenEnv.git",
-    "openenv-core[core]>=0.2.2",
-    # Environment-specific dependencies
-    "openai>=1.0.0",
-]
-
-[project.optional-dependencies]
-dev = [
-    "pytest>=8.0.0",
-    "pytest-cov>=4.0.0",
-]
-
-[project.scripts]
-# Server entry point - enables running via: uv run --project . server
-# or: python -m Cyber_analyst.server.app
-server = "Cyber_analyst.server.app:main"
-
-[tool.setuptools]
-include-package-data = true
-packages = ["Cyber_analyst", "Cyber_analyst.server"]
-package-dir = { "Cyber_analyst" = ".", "Cyber_analyst.server" = "server" }
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+[build-system]
+requires = ["setuptools>=45", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "openenv-Cyber_analyst"
+version = "0.1.0"
+description = "Cyber Analyst environment for OpenEnv"
+requires-python = ">=3.10"
+dependencies = [
+    # Core OpenEnv runtime (provides FastAPI server + HTTP client types)
+    # install from github
+    # "openenv-core[core] @ git+https://github.com/meta-pytorch/OpenEnv.git",
+    "openenv-core[core]>=0.2.2",
+    # Environment-specific dependencies
+    "openai>=1.0.0",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=8.0.0",
+    "pytest-cov>=4.0.0",
+]
+
+[project.scripts]
+# Server entry point - enables running via: uv run --project . server
+# or: python -m Cyber_analyst.server.app
+server = "Cyber_analyst.server.app:main"
+
+[tool.setuptools]
+include-package-data = true
+packages = ["Cyber_analyst", "Cyber_analyst.server"]
+package-dir = { "Cyber_analyst" = ".", "Cyber_analyst.server" = "server" }

sample_inference_script.py

@@ -0,0 +1,188 @@
+"""
+Inference Script Example
+===================================
+MANDATORY
+- Before submitting, ensure the following variables are defined in your environment configuration:
+    API_BASE_URL   The API endpoint for the LLM.
+    MODEL_NAME     The model identifier to use for inference.
+    HF_TOKEN       Your Hugging Face / API key.
+    LOCAL_IMAGE_NAME The name of the local image to use for the environment if you are using from_docker_image()
+                     method
+
+- Defaults are set only for API_BASE_URL and MODEL_NAME 
+    (and should reflect your active inference setup):
+    API_BASE_URL = os.getenv("API_BASE_URL", "<your-active-endpoint>")
+    MODEL_NAME = os.getenv("MODEL_NAME", "<your-active-model>")
+    
+- The inference script must be named `inference.py` and placed in the root directory of the project
+- Participants must use OpenAI Client for all LLM calls using above variables
+
+STDOUT FORMAT
+- The script must emit exactly three line types to stdout, in this order:
+
+    [START] task=<task_name> env=<benchmark> model=<model_name>
+    [STEP]  step=<n> action=<action_str> reward=<0.00> done=<true|false> error=<msg|null>
+    [END]   success=<true|false> steps=<n> score=<score> rewards=<r1,r2,...,rn>
+
+  Rules:
+    - One [START] line at episode begin.
+    - One [STEP] line per step, immediately after env.step() returns.
+    - One [END] line after env.close(), always emitted (even on exception).
+    - reward and rewards are formatted to 2 decimal places.
+    - done and success are lowercase booleans: true or false.
+    - error is the raw last_action_error string, or null if none.
+    - All fields on a single line with no newlines within a line.
+    - Each tasks should return score in [0, 1]
+
+  Example:
+    [START] task=click-test env=miniwob model=Qwen3-VL-30B
+    [STEP] step=1 action=click('123') reward=0.00 done=false error=null
+    [STEP] step=2 action=fill('456','text') reward=0.00 done=false error=null
+    [STEP] step=3 action=click('789') reward=1.00 done=true error=null
+    [END] success=true steps=3 score=1.00 rewards=0.00,0.00,1.00
+"""
+
+import asyncio
+import os
+import textwrap
+from typing import List, Optional
+
+from openai import OpenAI
+
+from my_env_v4 import MyEnvV4Action, MyEnvV4Env
+IMAGE_NAME = os.getenv("IMAGE_NAME") # If you are using docker image 
+API_KEY = os.getenv("HF_TOKEN") or os.getenv("API_KEY")
+
+API_BASE_URL = os.getenv("API_BASE_URL") or "https://router.huggingface.co/v1"
+MODEL_NAME = os.getenv("MODEL_NAME") or "Qwen/Qwen2.5-72B-Instruct"
+TASK_NAME = os.getenv("MY_ENV_V4_TASK", "echo")
+BENCHMARK = os.getenv("MY_ENV_V4_BENCHMARK", "my_env_v4")
+MAX_STEPS = 8
+TEMPERATURE = 0.7
+MAX_TOKENS = 150
+SUCCESS_SCORE_THRESHOLD = 0.1  # normalized score in [0, 1]
+
+# Max possible reward: each token contributes 0.1, across all steps
+_MAX_REWARD_PER_STEP = MAX_TOKENS * 0.1
+MAX_TOTAL_REWARD = MAX_STEPS * _MAX_REWARD_PER_STEP
+
+SYSTEM_PROMPT = textwrap.dedent(
+    """
+    You are interacting with a simple echo environment.
+    Each turn you must send a message. The environment will echo it back.
+    Reward is proportional to message length: reward = len(message) * 0.1
+    Your goal is to maximize total reward by sending meaningful, substantive messages.
+    Reply with exactly one message string — no quotes, no prefixes, just the message text.
+    """
+).strip()
+
+
+def log_start(task: str, env: str, model: str) -> None:
+    print(f"[START] task={task} env={env} model={model}", flush=True)
+
+
+def log_step(step: int, action: str, reward: float, done: bool, error: Optional[str]) -> None:
+    error_val = error if error else "null"
+    done_val = str(done).lower()
+    print(
+        f"[STEP] step={step} action={action} reward={reward:.2f} done={done_val} error={error_val}",
+        flush=True,
+    )
+
+
+def log_end(success: bool, steps: int, score: float, rewards: List[float]) -> None:
+    rewards_str = ",".join(f"{r:.2f}" for r in rewards)
+    print(f"[END] success={str(success).lower()} steps={steps} score={score:.3f} rewards={rewards_str}", flush=True)
+
+
+def build_user_prompt(step: int, last_echoed: str, last_reward: float, history: List[str]) -> str:
+    history_block = "\n".join(history[-4:]) if history else "None"
+    return textwrap.dedent(
+        f"""
+        Step: {step}
+        Last echoed message: {last_echoed!r}
+        Last reward: {last_reward:.2f}
+        Previous steps:
+        {history_block}
+        Send your next message.
+        """
+    ).strip()
+
+
+def get_model_message(client: OpenAI, step: int, last_echoed: str, last_reward: float, history: List[str]) -> str:
+    user_prompt = build_user_prompt(step, last_echoed, last_reward, history)
+    try:
+        completion = client.chat.completions.create(
+            model=MODEL_NAME,
+            messages=[
+                {"role": "system", "content": SYSTEM_PROMPT},
+                {"role": "user", "content": user_prompt},
+            ],
+            temperature=TEMPERATURE,
+            max_tokens=MAX_TOKENS,
+            stream=False,
+        )
+        text = (completion.choices[0].message.content or "").strip()
+        return text if text else "hello"
+    except Exception as exc:
+        print(f"[DEBUG] Model request failed: {exc}", flush=True)
+        return "hello"
+
+
+async def main() -> None:
+    client = OpenAI(base_url=API_BASE_URL, api_key=API_KEY)
+
+    env = await MyEnvV4Env.from_docker_image(IMAGE_NAME)
+
+    history: List[str] = []
+    rewards: List[float] = []
+    steps_taken = 0
+    score = 0.0
+    success = False
+
+    log_start(task=TASK_NAME, env=BENCHMARK, model=MODEL_NAME)
+
+    try:
+        result = await env.reset() # OpenENV.reset()
+        last_echoed = result.observation.echoed_message
+        last_reward = 0.0
+
+        for step in range(1, MAX_STEPS + 1):
+            if result.done:
+                break
+
+            message = get_model_message(client, step, last_echoed, last_reward, history)
+
+            result = await env.step(MyEnvV4Action(message=message))
+            obs = result.observation
+
+            reward = result.reward or 0.0
+            done = result.done
+            error = None
+
+            rewards.append(reward)
+            steps_taken = step
+            last_echoed = obs.echoed_message
+            last_reward = reward
+
+            log_step(step=step, action=message, reward=reward, done=done, error=error)
+
+            history.append(f"Step {step}: {message!r} -> reward {reward:+.2f}")
+
+            if done:
+                break
+
+        score = sum(rewards) / MAX_TOTAL_REWARD if MAX_TOTAL_REWARD > 0 else 0.0
+        score = min(max(score, 0.0), 1.0)  # clamp to [0, 1]
+        success = score >= SUCCESS_SCORE_THRESHOLD
+
+    finally:
+        try:
+            await env.close()
+        except Exception as e:
+            print(f"[DEBUG] env.close() error (container cleanup): {e}", flush=True)
+        log_end(success=success, steps=steps_taken, score=score, rewards=rewards)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

scripts/validate-submission.sh

@@ -0,0 +1,188 @@
+#!/usr/bin/env bash
+#
+# validate-submission.sh — OpenEnv Submission Validator
+#
+# Checks that your HF Space is live, Docker image builds, and openenv validate passes.
+#
+# Prerequisites:
+#   - Docker:       https://docs.docker.com/get-docker/
+#   - openenv-core: pip install openenv-core
+#   - curl (usually pre-installed)
+#
+# Run:
+#   curl -fsSL https://raw.githubusercontent.com/<owner>/<repo>/main/scripts/validate-submission.sh | bash -s -- <ping_url> [repo_dir]
+#
+#   Or download and run locally:
+#     chmod +x validate-submission.sh
+#     ./validate-submission.sh <ping_url> [repo_dir]
+#
+# Arguments:
+#   ping_url   Your HuggingFace Space URL (e.g. https://your-space.hf.space)
+#   repo_dir   Path to your repo (default: current directory)
+#
+# Examples:
+#   ./validate-submission.sh https://my-team.hf.space
+#   ./validate-submission.sh https://my-team.hf.space ./my-repo
+#
+
+set -uo pipefail
+
+DOCKER_BUILD_TIMEOUT=600
+if [ -t 1 ]; then
+  RED='\033[0;31m'
+  GREEN='\033[0;32m'
+  YELLOW='\033[1;33m'
+  BOLD='\033[1m'
+  NC='\033[0m'
+else
+  RED='' GREEN='' YELLOW='' BOLD='' NC=''
+fi
+
+run_with_timeout() {
+  local secs="$1"; shift
+  if command -v timeout &>/dev/null; then
+    timeout "$secs" "$@"
+  elif command -v gtimeout &>/dev/null; then
+    gtimeout "$secs" "$@"
+  else
+    "$@" &
+    local pid=$!
+    ( sleep "$secs" && kill "$pid" 2>/dev/null ) &
+    local watcher=$!
+    wait "$pid" 2>/dev/null
+    local rc=$?
+    kill "$watcher" 2>/dev/null
+    wait "$watcher" 2>/dev/null
+    return $rc
+  fi
+}
+
+portable_mktemp() {
+  local prefix="${1:-validate}"
+  mktemp "${TMPDIR:-/tmp}/${prefix}-XXXXXX" 2>/dev/null || mktemp
+}
+
+CLEANUP_FILES=()
+cleanup() { rm -f "${CLEANUP_FILES[@]+"${CLEANUP_FILES[@]}"}"; }
+trap cleanup EXIT
+
+PING_URL="${1:-}"
+REPO_DIR="${2:-.}"
+
+if [ -z "$PING_URL" ]; then
+  printf "Usage: %s <ping_url> [repo_dir]\n" "$0"
+  printf "\n"
+  printf "  ping_url   Your HuggingFace Space URL (e.g. https://your-space.hf.space)\n"
+  printf "  repo_dir   Path to your repo (default: current directory)\n"
+  exit 1
+fi
+
+if ! REPO_DIR="$(cd "$REPO_DIR" 2>/dev/null && pwd)"; then
+  printf "Error: directory '%s' not found\n" "${2:-.}"
+  exit 1
+fi
+PING_URL="${PING_URL%/}"
+export PING_URL
+PASS=0
+
+log()  { printf "[%s] %b\n" "$(date -u +%H:%M:%S)" "$*"; }
+pass() { log "${GREEN}PASSED${NC} -- $1"; PASS=$((PASS + 1)); }
+fail() { log "${RED}FAILED${NC} -- $1"; }
+hint() { printf "  ${YELLOW}Hint:${NC} %b\n" "$1"; }
+stop_at() {
+  printf "\n"
+  printf "${RED}${BOLD}Validation stopped at %s.${NC} Fix the above before continuing.\n" "$1"
+  exit 1
+}
+
+printf "\n"
+printf "${BOLD}========================================${NC}\n"
+printf "${BOLD}  OpenEnv Submission Validator${NC}\n"
+printf "${BOLD}========================================${NC}\n"
+log "Repo:     $REPO_DIR"
+log "Ping URL: $PING_URL"
+printf "\n"
+
+log "${BOLD}Step 1/3: Pinging HF Space${NC} ($PING_URL/reset) ..."
+
+CURL_OUTPUT=$(portable_mktemp "validate-curl")
+CLEANUP_FILES+=("$CURL_OUTPUT")
+HTTP_CODE=$(curl -s -o "$CURL_OUTPUT" -w "%{http_code}" -X POST \
+  -H "Content-Type: application/json" -d '{}' \
+  "$PING_URL/reset" --max-time 30 2>"$CURL_OUTPUT" || printf "000")
+
+if [ "$HTTP_CODE" = "200" ]; then
+  pass "HF Space is live and responds to /reset"
+elif [ "$HTTP_CODE" = "000" ]; then
+  fail "HF Space not reachable (connection failed or timed out)"
+  hint "Check your network connection and that the Space is running."
+  hint "Try: curl -s -o /dev/null -w '%%{http_code}' -X POST $PING_URL/reset"
+  stop_at "Step 1"
+else
+  fail "HF Space /reset returned HTTP $HTTP_CODE (expected 200)"
+  hint "Make sure your Space is running and the URL is correct."
+  hint "Try opening $PING_URL in your browser first."
+  stop_at "Step 1"
+fi
+
+log "${BOLD}Step 2/3: Running docker build${NC} ..."
+
+if ! command -v docker &>/dev/null; then
+  fail "docker command not found"
+  hint "Install Docker: https://docs.docker.com/get-docker/"
+  stop_at "Step 2"
+fi
+
+if [ -f "$REPO_DIR/Dockerfile" ]; then
+  DOCKER_CONTEXT="$REPO_DIR"
+  DOCKERFILE="$REPO_DIR/Dockerfile"
+elif [ -f "$REPO_DIR/server/Dockerfile" ]; then
+  DOCKER_CONTEXT="$REPO_DIR"
+  DOCKERFILE="$REPO_DIR/server/Dockerfile"
+else
+  fail "No Dockerfile found in repo root or server/ directory"
+  stop_at "Step 2"
+fi
+
+log "  Found Dockerfile at $DOCKERFILE"
+log "  Docker context: $DOCKER_CONTEXT"
+
+BUILD_OK=false
+BUILD_OUTPUT=$(run_with_timeout "$DOCKER_BUILD_TIMEOUT" docker build -f "$DOCKERFILE" "$DOCKER_CONTEXT" 2>&1) && BUILD_OK=true
+
+if [ "$BUILD_OK" = true ]; then
+  pass "Docker build succeeded"
+else
+  fail "Docker build failed (timeout=${DOCKER_BUILD_TIMEOUT}s)"
+  printf "%s\n" "$BUILD_OUTPUT" | tail -20
+  stop_at "Step 2"
+fi
+
+log "${BOLD}Step 3/3: Running openenv validate${NC} ..."
+
+if ! command -v openenv &>/dev/null; then
+  fail "openenv command not found"
+  hint "Install it: pip install openenv-core"
+  stop_at "Step 3"
+fi
+
+VALIDATE_OK=false
+VALIDATE_OUTPUT=$(cd "$REPO_DIR" && openenv validate 2>&1) && VALIDATE_OK=true
+
+if [ "$VALIDATE_OK" = true ]; then
+  pass "openenv validate passed"
+  [ -n "$VALIDATE_OUTPUT" ] && log "  $VALIDATE_OUTPUT"
+else
+  fail "openenv validate failed"
+  printf "%s\n" "$VALIDATE_OUTPUT"
+  stop_at "Step 3"
+fi
+
+printf "\n"
+printf "${BOLD}========================================${NC}\n"
+printf "${GREEN}${BOLD}  All 3/3 checks passed!${NC}\n"
+printf "${GREEN}${BOLD}  Your submission is ready to submit.${NC}\n"
+printf "${BOLD}========================================${NC}\n"
+printf "\n"
+
+exit 0

server/Cyber_analyst_environment.py

@@ -1,506 +1,506 @@
-# Copyright (c) Meta Platforms, Inc. and affiliates.
-# All rights reserved.
-#
-# This source code is licensed under the BSD-style license found in the
-# LICENSE file in the root directory of this source tree.
-
-"""SecOps Evidence Gym environment implementation."""
-
-from __future__ import annotations
-
-import hashlib
-import json
-from collections import Counter
-from typing import Any
-from uuid import uuid4
-
-from openenv.core.env_server.interfaces import Environment
-
-try:
-    from ..models import (
-        CyberAnalystAction,
-        CyberAnalystObservation,
-        CyberAnalystState,
-    )
-    from .graders import safe_reward, score_report
-    from .tasks import DEFAULT_TASK_ID, TOOL_CATALOG, build_scenario
-except ImportError:  # pragma: no cover - supports direct module execution
-    from models import CyberAnalystAction, CyberAnalystObservation, CyberAnalystState
-    from server.graders import safe_reward, score_report
-    from server.tasks import DEFAULT_TASK_ID, TOOL_CATALOG, build_scenario
-
-
-class CyberAnalystEnvironment(
-    Environment[CyberAnalystAction, CyberAnalystObservation, CyberAnalystState]
-):
-    """A safe, deterministic evidence-grounded cyber analyst benchmark."""
-
-    SUPPORTS_CONCURRENT_SESSIONS: bool = True
-    MAX_STEPS = 12
-    REPEAT_HARD_STOP = 6
-
-    def __init__(self):
-        super().__init__()
-        self._scenario: dict[str, Any] = {}
-        self._state = CyberAnalystState()
-        self._discovered_evidence: set[str] = set()
-        self._candidate_findings: dict[str, dict[str, Any]] = {}
-        self._verified_findings: list[dict[str, Any]] = []
-        self._validated_finding_ids: set[str] = set()
-        self._action_counts: Counter[str] = Counter()
-        self._last_score_breakdown: dict[str, Any] = {}
-        self._trajectory_events: list[dict[str, Any]] = []
-        self._initialize_episode(DEFAULT_TASK_ID, seed=None, episode_id=None)
-
-    def reset(
-        self,
-        seed: int | None = None,
-        episode_id: str | None = None,
-        task_id: str = DEFAULT_TASK_ID,
-        **_: Any,
-    ) -> CyberAnalystObservation:
-        """Reset the selected deterministic task."""
-
-        self._initialize_episode(task_id=task_id, seed=seed, episode_id=episode_id)
-        tool_result = {
-            "message": "Cyber Analyst environment ready.",
-            "allowed_scope": "Synthetic artifacts only. No live targets or shell.",
-        }
-        obs = self._observation(
-            tool_result={
-                **tool_result,
-                "trajectory_jsonl": self.export_trajectory_jsonl(),
-            },
-            reward=0.01,
-        )
-        self._record_trajectory("reset", None, tool_result, obs.reward, obs.done, obs.error)
-        return obs
-
-    def step(  # type: ignore[override]
-        self,
-        action: CyberAnalystAction,
-        timeout_s: float | None = None,
-        **_: Any,
-    ) -> CyberAnalystObservation:
-        """Execute one bounded simulator tool call."""
-
-        del timeout_s
-
-        if self._state.done:
-            tool_result = {"message": "Episode is already complete."}
-            obs = self._observation(
-                tool_result=tool_result,
-                reward=0.01,
-                done=True,
-                error="episode_already_done",
-            )
-            self._record_trajectory("step", action, tool_result, obs.reward, obs.done, obs.error)
-            return obs
-
-        self._state.step_count += 1
-        self._state.step_budget_remaining = max(
-            0, self.MAX_STEPS - self._state.step_count
-        )
-
-        signature = self._action_signature(action)
-        self._action_counts[signature] += 1
-        repeat_count = self._action_counts[signature]
-
-        if repeat_count >= self.REPEAT_HARD_STOP:
-            self._state.phase = "done"
-            self._state.done = True
-            self._last_score_breakdown = {
-                "score": 0.03,
-                "repeat_hard_stop": True,
-                "signature": signature,
-            }
-            tool_result = {"message": "Episode stopped after repeated identical actions."}
-            obs = self._observation(
-                tool_result=tool_result,
-                reward=0.03,
-                done=True,
-                error="repeat_hard_stop",
-            )
-            self._record_trajectory("step", action, tool_result, obs.reward, obs.done, obs.error)
-            return obs
-
-        handler = getattr(self, f"_tool_{action.tool_name}", None)
-        if handler is None:
-            tool_result = {
-                "ok": False,
-                "message": f"Unsupported tool: {action.tool_name}",
-                "available_tools": [tool["name"] for tool in TOOL_CATALOG],
-            }
-            obs = self._step_observation(
-                tool_result=tool_result,
-                repeat_count=repeat_count,
-                error="unsupported_tool",
-            )
-            self._record_trajectory("step", action, tool_result, obs.reward, obs.done, obs.error)
-            return obs
-
-        try:
-            result, reward_delta, done = handler(action.args)
-            error = ""
-        except Exception as exc:  # pragma: no cover - defensive rollout guard
-            result = {"ok": False, "message": str(exc)}
-            reward_delta = -0.05
-            done = False
-            error = exc.__class__.__name__
-
-        if self._state.step_budget_remaining <= 0 and not done:
-            done = True
-            self._state.phase = "done"
-            self._state.done = True
-            result = {
-                **result,
-                "timeout": True,
-                "message": "Step budget exhausted before report submission.",
-            }
-            reward_delta -= 0.10
-
-        obs = self._step_observation(
-            tool_result=result,
-            repeat_count=repeat_count,
-            reward_delta=reward_delta,
-            done=done,
-            error=error,
-        )
-        self._record_trajectory("step", action, result, obs.reward, obs.done, obs.error)
-        return obs
-
-    @property
-    def state(self) -> CyberAnalystState:
-        """Return the current episode state summary."""
-
-        return self._state
-
-    def _initialize_episode(
-        self, task_id: str, seed: int | None, episode_id: str | None
-    ) -> None:
-        self._scenario = build_scenario(task_id, seed)
-        self._discovered_evidence = set()
-        self._candidate_findings = {}
-        self._verified_findings = []
-        self._validated_finding_ids = set()
-        self._action_counts = Counter()
-        self._last_score_breakdown = {}
-        self._trajectory_events = []
-        self._state = CyberAnalystState(
-            episode_id=episode_id or str(uuid4()),
-            step_count=0,
-            task_id=self._scenario["task_id"],
-            seed=seed,
-            phase="investigate",
-            step_budget_remaining=self.MAX_STEPS,
-            recent_evidence_ids=[],
-            verified_finding_ids=[],
-            done=False,
-        )
-
-    def export_trajectory_jsonl(self) -> str:
-        """Return the current episode trajectory as JSONL for offline analysis."""
-
-        return "\n".join(
-            json.dumps(event, sort_keys=True, default=str)
-            for event in self._trajectory_events
-        )
-
-    def _record_trajectory(
-        self,
-        event_type: str,
-        action: CyberAnalystAction | None,
-        tool_result: dict[str, Any],
-        reward: float | int | None,
-        done: bool,
-        error: str,
-    ) -> None:
-        action_payload = None
-        if action is not None:
-            action_payload = action.model_dump(exclude_none=True)
-        self._trajectory_events.append(
-            {
-                "episode_id": self._state.episode_id,
-                "task_id": self._state.task_id,
-                "seed": self._state.seed,
-                "event_type": event_type,
-                "step": self._state.step_count,
-                "phase": self._state.phase,
-                "action": action_payload,
-                "tool_result": tool_result,
-                "evidence_ids": sorted(self._discovered_evidence),
-                "verified_finding_ids": list(self._state.verified_finding_ids),
-                "reward": reward,
-                "done": done,
-                "error": error,
-            }
-        )
-
-    def _observation(
-        self,
-        tool_result: dict[str, Any] | None = None,
-        reward: float = 0.01,
-        done: bool | None = None,
-        error: str = "",
-    ) -> CyberAnalystObservation:
-        done_value = self._state.done if done is None else done
-        return CyberAnalystObservation(
-            task_id=self._scenario.get("task_id", ""),
-            alert=self._scenario.get("alert", ""),
-            phase=self._state.phase,
-            tool_catalog=TOOL_CATALOG,
-            tool_result=tool_result or {},
-            evidence_ids=sorted(self._discovered_evidence),
-            verified_findings=list(self._verified_findings),
-            candidate_findings=list(self._candidate_findings.values()),
-            step_budget_remaining=self._state.step_budget_remaining,
-            score_breakdown=dict(self._last_score_breakdown),
-            error=error,
-            done=done_value,
-            reward=safe_reward(reward),
-        )
-
-    def _step_observation(
-        self,
-        tool_result: dict[str, Any],
-        repeat_count: int,
-        reward_delta: float = 0.0,
-        done: bool = False,
-        error: str = "",
-    ) -> CyberAnalystObservation:
-        reward = 0.04 + reward_delta - 0.01
-        if repeat_count > 2:
-            reward -= 0.03 * (repeat_count - 2)
-
-        if done:
-            self._state.phase = "done"
-            self._state.done = True
-
-        self._state.recent_evidence_ids = sorted(self._discovered_evidence)[-5:]
-        self._state.verified_finding_ids = [
-            finding["finding_id"] for finding in self._verified_findings
-        ]
-
-        return self._observation(
-            tool_result=tool_result,
-            reward=safe_reward(reward),
-            done=self._state.done,
-            error=error,
-        )
-
-    def _action_signature(self, action: CyberAnalystAction) -> str:
-        payload = {
-            "tool_name": action.tool_name,
-            "args": action.args,
-        }
-        encoded = json.dumps(payload, sort_keys=True, default=str)
-        return hashlib.sha256(encoded.encode("utf-8")).hexdigest()[:16]
-
-    def _record_evidence(self, evidence_ids: list[str]) -> int:
-        relevant = set(self._scenario.get("required_evidence", [])) | set(
-            self._scenario.get("supporting_evidence", [])
-        )
-        new_relevant = 0
-        for evidence_id in evidence_ids:
-            if evidence_id not in self._discovered_evidence and evidence_id in relevant:
-                new_relevant += 1
-            self._discovered_evidence.add(evidence_id)
-        return new_relevant
-
-    def _filter_entries(
-        self, entries: list[dict[str, Any]], service_id: str = "", query: str = ""
-    ) -> list[dict[str, Any]]:
-        normalized_service = self._resolve_service_id(service_id).lower()
-        normalized_query = query.strip().lower()
-        matches: list[dict[str, Any]] = []
-        for entry in entries:
-            service_matches = (
-                not normalized_service
-                or str(entry.get("service_id", "")).lower() == normalized_service
-            )
-            search_blob = " ".join(
-                [
-                    str(entry.get("text", "")),
-                    str(entry.get("source", "")),
-                    " ".join(str(tag) for tag in entry.get("tags", [])),
-                ]
-            ).lower()
-            query_matches = not normalized_query or normalized_query in search_blob
-            if service_matches and query_matches:
-                matches.append(entry)
-        return matches
-
-    def _resolve_service_id(self, service_id: str) -> str:
-        normalized = service_id.strip()
-        aliases = self._scenario.get("service_aliases", {})
-        return str(aliases.get(normalized, normalized))
-
-    def _evidence_payload(self, entries: list[dict[str, Any]]) -> dict[str, Any]:
-        evidence_ids = [entry["evidence_id"] for entry in entries]
-        new_relevant = self._record_evidence(evidence_ids)
-        return {
-            "ok": True,
-            "evidence_ids": evidence_ids,
-            "new_relevant_evidence": new_relevant,
-            "entries": [
-                {
-                    "evidence_id": entry["evidence_id"],
-                    "service_id": entry.get("service_id", ""),
-                    "source": entry.get("source", ""),
-                    "text": entry.get("text", ""),
-                }
-                for entry in entries
-            ],
-        }
-
-    def _tool_list_assets(self, args: dict[str, Any]) -> tuple[dict[str, Any], float, bool]:
-        del args
-        return {"ok": True, "assets": self._scenario["assets"]}, 0.0, False
-
-    def _tool_get_log_events(
-        self, args: dict[str, Any]
-    ) -> tuple[dict[str, Any], float, bool]:
-        entries = self._filter_entries(
-            self._scenario.get("logs", []),
-            service_id=str(args.get("service_id", "")),
-            query=str(args.get("query", "")),
-        )
-        payload = self._evidence_payload(entries)
-        return payload, 0.02 * payload["new_relevant_evidence"], False
-
-    def _tool_check_security_headers(
-        self, args: dict[str, Any]
-    ) -> tuple[dict[str, Any], float, bool]:
-        requested_service = self._resolve_service_id(str(args.get("service_id", ""))).lower()
-        snapshots = self._scenario.get("headers", {})
-        results = []
-        evidence_ids = []
-        for service_id, snapshot in snapshots.items():
-            if requested_service and service_id.lower() != requested_service:
-                continue
-            evidence_ids.append(snapshot["evidence_id"])
-            results.append(
-                {
-                    "service_id": service_id,
-                    "evidence_id": snapshot["evidence_id"],
-                    "present": snapshot.get("present", []),
-                    "missing": snapshot.get("missing", []),
-                    "passed": not snapshot.get("missing"),
-                }
-            )
-        new_relevant = self._record_evidence(evidence_ids)
-        return (
-            {
-                "ok": True,
-                "evidence_ids": evidence_ids,
-                "new_relevant_evidence": new_relevant,
-                "header_results": results,
-            },
-            0.02 * new_relevant,
-            False,
-        )
-
-    def _tool_search_repo(self, args: dict[str, Any]) -> tuple[dict[str, Any], float, bool]:
-        entries = self._filter_entries(
-            self._scenario.get("repo", []), query=str(args.get("query", ""))
-        )
-        payload = self._evidence_payload(entries)
-        return payload, 0.02 * payload["new_relevant_evidence"], False
-
-    def _tool_scan_dependencies(
-        self, args: dict[str, Any]
-    ) -> tuple[dict[str, Any], float, bool]:
-        del args
-        payload = self._evidence_payload(self._scenario.get("dependencies", []))
-        return payload, 0.02 * payload["new_relevant_evidence"], False
-
-    def _tool_create_finding(
-        self, args: dict[str, Any]
-    ) -> tuple[dict[str, Any], float, bool]:
-        evidence_ids = args.get("evidence_ids", [])
-        if isinstance(evidence_ids, str):
-            evidence_ids = [evidence_ids]
-        evidence_ids = [str(evidence_id) for evidence_id in evidence_ids]
-
-        finding_id = f"FND-{len(self._candidate_findings) + 1:03d}"
-        finding = {
-            "finding_id": finding_id,
-            "finding_type": str(args.get("finding_type", "")),
-            "evidence_ids": evidence_ids,
-            "severity_guess": str(args.get("severity_guess", "")),
-            "remediation": str(args.get("remediation", "")),
-            "validated": False,
-            "matching_gt_id": None,
-        }
-        self._candidate_findings[finding_id] = finding
-
-        well_formed = bool(
-            finding["finding_type"] and evidence_ids and finding["remediation"]
-        )
-        return (
-            {"ok": True, "finding_id": finding_id, "finding": finding},
-            0.03 if well_formed else 0.0,
-            False,
-        )
-
-    def _tool_validate_finding(
-        self, args: dict[str, Any]
-    ) -> tuple[dict[str, Any], float, bool]:
-        finding_id = str(args.get("finding_id", ""))
-        finding = self._candidate_findings.get(finding_id)
-        if finding is None:
-            return (
-                {"ok": False, "message": f"Unknown finding_id: {finding_id}"},
-                -0.03,
-                False,
-            )
-
-        expected_type = self._scenario["finding_type"]
-        required_evidence = set(self._scenario.get("required_evidence", []))
-        supplied_evidence = set(finding.get("evidence_ids", []))
-        verified = (
-            finding.get("finding_type") == expected_type
-            and bool(required_evidence & supplied_evidence)
-        )
-        self._validated_finding_ids.add(finding_id)
-        finding["validated"] = verified
-        finding["matching_gt_id"] = self._scenario["ground_truth_id"] if verified else None
-
-        if verified and not any(
-            item["finding_id"] == finding_id for item in self._verified_findings
-        ):
-            self._verified_findings.append(dict(finding))
-
-        return (
-            {
-                "ok": True,
-                "finding_id": finding_id,
-                "verified": verified,
-                "matching_gt_id": finding["matching_gt_id"],
-            },
-            0.08 if verified else -0.02,
-            False,
-        )
-
-    def _tool_submit_report(
-        self, args: dict[str, Any]
-    ) -> tuple[dict[str, Any], float, bool]:
-        report = args.get("report_json", {})
-        score, breakdown = score_report(
-            self._scenario["task_id"],
-            report,
-            verified_findings=self._verified_findings,
-            validation_attempted=bool(self._validated_finding_ids),
-        )
-        self._last_score_breakdown = breakdown
-        return (
-            {
-                "ok": True,
-                "submitted": True,
-                "score": score,
-                "score_breakdown": breakdown,
-                "trajectory_jsonl": self.export_trajectory_jsonl(),
-            },
-            score,
-            True,
-        )
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+"""SecOps Evidence Gym environment implementation."""
+
+from __future__ import annotations
+
+import hashlib
+import json
+from collections import Counter
+from typing import Any
+from uuid import uuid4
+
+from openenv.core.env_server.interfaces import Environment
+
+try:
+    from ..models import (
+        CyberAnalystAction,
+        CyberAnalystObservation,
+        CyberAnalystState,
+    )
+    from .graders import safe_reward, score_report
+    from .tasks import DEFAULT_TASK_ID, TOOL_CATALOG, build_scenario
+except ImportError:  # pragma: no cover - supports direct module execution
+    from models import CyberAnalystAction, CyberAnalystObservation, CyberAnalystState
+    from server.graders import safe_reward, score_report
+    from server.tasks import DEFAULT_TASK_ID, TOOL_CATALOG, build_scenario
+
+
+class CyberAnalystEnvironment(
+    Environment[CyberAnalystAction, CyberAnalystObservation, CyberAnalystState]
+):
+    """A safe, deterministic evidence-grounded cyber analyst benchmark."""
+
+    SUPPORTS_CONCURRENT_SESSIONS: bool = True
+    MAX_STEPS = 12
+    REPEAT_HARD_STOP = 6
+
+    def __init__(self):
+        super().__init__()
+        self._scenario: dict[str, Any] = {}
+        self._state = CyberAnalystState()
+        self._discovered_evidence: set[str] = set()
+        self._candidate_findings: dict[str, dict[str, Any]] = {}
+        self._verified_findings: list[dict[str, Any]] = []
+        self._validated_finding_ids: set[str] = set()
+        self._action_counts: Counter[str] = Counter()
+        self._last_score_breakdown: dict[str, Any] = {}
+        self._trajectory_events: list[dict[str, Any]] = []
+        self._initialize_episode(DEFAULT_TASK_ID, seed=None, episode_id=None)
+
+    def reset(
+        self,
+        seed: int | None = None,
+        episode_id: str | None = None,
+        task_id: str = DEFAULT_TASK_ID,
+        **_: Any,
+    ) -> CyberAnalystObservation:
+        """Reset the selected deterministic task."""
+
+        self._initialize_episode(task_id=task_id, seed=seed, episode_id=episode_id)
+        tool_result = {
+            "message": "Cyber Analyst environment ready.",
+            "allowed_scope": "Synthetic artifacts only. No live targets or shell.",
+        }
+        obs = self._observation(
+            tool_result={
+                **tool_result,
+                "trajectory_jsonl": self.export_trajectory_jsonl(),
+            },
+            reward=0.01,
+        )
+        self._record_trajectory("reset", None, tool_result, obs.reward, obs.done, obs.error)
+        return obs
+
+    def step(  # type: ignore[override]
+        self,
+        action: CyberAnalystAction,
+        timeout_s: float | None = None,
+        **_: Any,
+    ) -> CyberAnalystObservation:
+        """Execute one bounded simulator tool call."""
+
+        del timeout_s
+
+        if self._state.done:
+            tool_result = {"message": "Episode is already complete."}
+            obs = self._observation(
+                tool_result=tool_result,
+                reward=0.01,
+                done=True,
+                error="episode_already_done",
+            )
+            self._record_trajectory("step", action, tool_result, obs.reward, obs.done, obs.error)
+            return obs
+
+        self._state.step_count += 1
+        self._state.step_budget_remaining = max(
+            0, self.MAX_STEPS - self._state.step_count
+        )
+
+        signature = self._action_signature(action)
+        self._action_counts[signature] += 1
+        repeat_count = self._action_counts[signature]
+
+        if repeat_count >= self.REPEAT_HARD_STOP:
+            self._state.phase = "done"
+            self._state.done = True
+            self._last_score_breakdown = {
+                "score": 0.03,
+                "repeat_hard_stop": True,
+                "signature": signature,
+            }
+            tool_result = {"message": "Episode stopped after repeated identical actions."}
+            obs = self._observation(
+                tool_result=tool_result,
+                reward=0.03,
+                done=True,
+                error="repeat_hard_stop",
+            )
+            self._record_trajectory("step", action, tool_result, obs.reward, obs.done, obs.error)
+            return obs
+
+        handler = getattr(self, f"_tool_{action.tool_name}", None)
+        if handler is None:
+            tool_result = {
+                "ok": False,
+                "message": f"Unsupported tool: {action.tool_name}",
+                "available_tools": [tool["name"] for tool in TOOL_CATALOG],
+            }
+            obs = self._step_observation(
+                tool_result=tool_result,
+                repeat_count=repeat_count,
+                error="unsupported_tool",
+            )
+            self._record_trajectory("step", action, tool_result, obs.reward, obs.done, obs.error)
+            return obs
+
+        try:
+            result, reward_delta, done = handler(action.args)
+            error = ""
+        except Exception as exc:  # pragma: no cover - defensive rollout guard
+            result = {"ok": False, "message": str(exc)}
+            reward_delta = -0.05
+            done = False
+            error = exc.__class__.__name__
+
+        if self._state.step_budget_remaining <= 0 and not done:
+            done = True
+            self._state.phase = "done"
+            self._state.done = True
+            result = {
+                **result,
+                "timeout": True,
+                "message": "Step budget exhausted before report submission.",
+            }
+            reward_delta -= 0.10
+
+        obs = self._step_observation(
+            tool_result=result,
+            repeat_count=repeat_count,
+            reward_delta=reward_delta,
+            done=done,
+            error=error,
+        )
+        self._record_trajectory("step", action, result, obs.reward, obs.done, obs.error)
+        return obs
+
+    @property
+    def state(self) -> CyberAnalystState:
+        """Return the current episode state summary."""
+
+        return self._state
+
+    def _initialize_episode(
+        self, task_id: str, seed: int | None, episode_id: str | None
+    ) -> None:
+        self._scenario = build_scenario(task_id, seed)
+        self._discovered_evidence = set()
+        self._candidate_findings = {}
+        self._verified_findings = []
+        self._validated_finding_ids = set()
+        self._action_counts = Counter()
+        self._last_score_breakdown = {}
+        self._trajectory_events = []
+        self._state = CyberAnalystState(
+            episode_id=episode_id or str(uuid4()),
+            step_count=0,
+            task_id=self._scenario["task_id"],
+            seed=seed,
+            phase="investigate",
+            step_budget_remaining=self.MAX_STEPS,
+            recent_evidence_ids=[],
+            verified_finding_ids=[],
+            done=False,
+        )
+
+    def export_trajectory_jsonl(self) -> str:
+        """Return the current episode trajectory as JSONL for offline analysis."""
+
+        return "\n".join(
+            json.dumps(event, sort_keys=True, default=str)
+            for event in self._trajectory_events
+        )
+
+    def _record_trajectory(
+        self,
+        event_type: str,
+        action: CyberAnalystAction | None,
+        tool_result: dict[str, Any],
+        reward: float | int | None,
+        done: bool,
+        error: str,
+    ) -> None:
+        action_payload = None
+        if action is not None:
+            action_payload = action.model_dump(exclude_none=True)
+        self._trajectory_events.append(
+            {
+                "episode_id": self._state.episode_id,
+                "task_id": self._state.task_id,
+                "seed": self._state.seed,
+                "event_type": event_type,
+                "step": self._state.step_count,
+                "phase": self._state.phase,
+                "action": action_payload,
+                "tool_result": tool_result,
+                "evidence_ids": sorted(self._discovered_evidence),
+                "verified_finding_ids": list(self._state.verified_finding_ids),
+                "reward": reward,
+                "done": done,
+                "error": error,
+            }
+        )
+
+    def _observation(
+        self,
+        tool_result: dict[str, Any] | None = None,
+        reward: float = 0.01,
+        done: bool | None = None,
+        error: str = "",
+    ) -> CyberAnalystObservation:
+        done_value = self._state.done if done is None else done
+        return CyberAnalystObservation(
+            task_id=self._scenario.get("task_id", ""),
+            alert=self._scenario.get("alert", ""),
+            phase=self._state.phase,
+            tool_catalog=TOOL_CATALOG,
+            tool_result=tool_result or {},
+            evidence_ids=sorted(self._discovered_evidence),
+            verified_findings=list(self._verified_findings),
+            candidate_findings=list(self._candidate_findings.values()),
+            step_budget_remaining=self._state.step_budget_remaining,
+            score_breakdown=dict(self._last_score_breakdown),
+            error=error,
+            done=done_value,
+            reward=safe_reward(reward),
+        )
+
+    def _step_observation(
+        self,
+        tool_result: dict[str, Any],
+        repeat_count: int,
+        reward_delta: float = 0.0,
+        done: bool = False,
+        error: str = "",
+    ) -> CyberAnalystObservation:
+        reward = 0.04 + reward_delta - 0.01
+        if repeat_count > 2:
+            reward -= 0.03 * (repeat_count - 2)
+
+        if done:
+            self._state.phase = "done"
+            self._state.done = True
+
+        self._state.recent_evidence_ids = sorted(self._discovered_evidence)[-5:]
+        self._state.verified_finding_ids = [
+            finding["finding_id"] for finding in self._verified_findings
+        ]
+
+        return self._observation(
+            tool_result=tool_result,
+            reward=safe_reward(reward),
+            done=self._state.done,
+            error=error,
+        )
+
+    def _action_signature(self, action: CyberAnalystAction) -> str:
+        payload = {
+            "tool_name": action.tool_name,
+            "args": action.args,
+        }
+        encoded = json.dumps(payload, sort_keys=True, default=str)
+        return hashlib.sha256(encoded.encode("utf-8")).hexdigest()[:16]
+
+    def _record_evidence(self, evidence_ids: list[str]) -> int:
+        relevant = set(self._scenario.get("required_evidence", [])) | set(
+            self._scenario.get("supporting_evidence", [])
+        )
+        new_relevant = 0
+        for evidence_id in evidence_ids:
+            if evidence_id not in self._discovered_evidence and evidence_id in relevant:
+                new_relevant += 1
+            self._discovered_evidence.add(evidence_id)
+        return new_relevant
+
+    def _filter_entries(
+        self, entries: list[dict[str, Any]], service_id: str = "", query: str = ""
+    ) -> list[dict[str, Any]]:
+        normalized_service = self._resolve_service_id(service_id).lower()
+        normalized_query = query.strip().lower()
+        matches: list[dict[str, Any]] = []
+        for entry in entries:
+            service_matches = (
+                not normalized_service
+                or str(entry.get("service_id", "")).lower() == normalized_service
+            )
+            search_blob = " ".join(
+                [
+                    str(entry.get("text", "")),
+                    str(entry.get("source", "")),
+                    " ".join(str(tag) for tag in entry.get("tags", [])),
+                ]
+            ).lower()
+            query_matches = not normalized_query or normalized_query in search_blob
+            if service_matches and query_matches:
+                matches.append(entry)
+        return matches
+
+    def _resolve_service_id(self, service_id: str) -> str:
+        normalized = service_id.strip()
+        aliases = self._scenario.get("service_aliases", {})
+        return str(aliases.get(normalized, normalized))
+
+    def _evidence_payload(self, entries: list[dict[str, Any]]) -> dict[str, Any]:
+        evidence_ids = [entry["evidence_id"] for entry in entries]
+        new_relevant = self._record_evidence(evidence_ids)
+        return {
+            "ok": True,
+            "evidence_ids": evidence_ids,
+            "new_relevant_evidence": new_relevant,
+            "entries": [
+                {
+                    "evidence_id": entry["evidence_id"],
+                    "service_id": entry.get("service_id", ""),
+                    "source": entry.get("source", ""),
+                    "text": entry.get("text", ""),
+                }
+                for entry in entries
+            ],
+        }
+
+    def _tool_list_assets(self, args: dict[str, Any]) -> tuple[dict[str, Any], float, bool]:
+        del args
+        return {"ok": True, "assets": self._scenario["assets"]}, 0.0, False
+
+    def _tool_get_log_events(
+        self, args: dict[str, Any]
+    ) -> tuple[dict[str, Any], float, bool]:
+        entries = self._filter_entries(
+            self._scenario.get("logs", []),
+            service_id=str(args.get("service_id", "")),
+            query=str(args.get("query", "")),
+        )
+        payload = self._evidence_payload(entries)
+        return payload, 0.02 * payload["new_relevant_evidence"], False
+
+    def _tool_check_security_headers(
+        self, args: dict[str, Any]
+    ) -> tuple[dict[str, Any], float, bool]:
+        requested_service = self._resolve_service_id(str(args.get("service_id", ""))).lower()
+        snapshots = self._scenario.get("headers", {})
+        results = []
+        evidence_ids = []
+        for service_id, snapshot in snapshots.items():
+            if requested_service and service_id.lower() != requested_service:
+                continue
+            evidence_ids.append(snapshot["evidence_id"])
+            results.append(
+                {
+                    "service_id": service_id,
+                    "evidence_id": snapshot["evidence_id"],
+                    "present": snapshot.get("present", []),
+                    "missing": snapshot.get("missing", []),
+                    "passed": not snapshot.get("missing"),
+                }
+            )
+        new_relevant = self._record_evidence(evidence_ids)
+        return (
+            {
+                "ok": True,
+                "evidence_ids": evidence_ids,
+                "new_relevant_evidence": new_relevant,
+                "header_results": results,
+            },
+            0.02 * new_relevant,
+            False,
+        )
+
+    def _tool_search_repo(self, args: dict[str, Any]) -> tuple[dict[str, Any], float, bool]:
+        entries = self._filter_entries(
+            self._scenario.get("repo", []), query=str(args.get("query", ""))
+        )
+        payload = self._evidence_payload(entries)
+        return payload, 0.02 * payload["new_relevant_evidence"], False
+
+    def _tool_scan_dependencies(
+        self, args: dict[str, Any]
+    ) -> tuple[dict[str, Any], float, bool]:
+        del args
+        payload = self._evidence_payload(self._scenario.get("dependencies", []))
+        return payload, 0.02 * payload["new_relevant_evidence"], False
+
+    def _tool_create_finding(
+        self, args: dict[str, Any]
+    ) -> tuple[dict[str, Any], float, bool]:
+        evidence_ids = args.get("evidence_ids", [])
+        if isinstance(evidence_ids, str):
+            evidence_ids = [evidence_ids]
+        evidence_ids = [str(evidence_id) for evidence_id in evidence_ids]
+
+        finding_id = f"FND-{len(self._candidate_findings) + 1:03d}"
+        finding = {
+            "finding_id": finding_id,
+            "finding_type": str(args.get("finding_type", "")),
+            "evidence_ids": evidence_ids,
+            "severity_guess": str(args.get("severity_guess", "")),
+            "remediation": str(args.get("remediation", "")),
+            "validated": False,
+            "matching_gt_id": None,
+        }
+        self._candidate_findings[finding_id] = finding
+
+        well_formed = bool(
+            finding["finding_type"] and evidence_ids and finding["remediation"]
+        )
+        return (
+            {"ok": True, "finding_id": finding_id, "finding": finding},
+            0.03 if well_formed else 0.0,
+            False,
+        )
+
+    def _tool_validate_finding(
+        self, args: dict[str, Any]
+    ) -> tuple[dict[str, Any], float, bool]:
+        finding_id = str(args.get("finding_id", ""))
+        finding = self._candidate_findings.get(finding_id)
+        if finding is None:
+            return (
+                {"ok": False, "message": f"Unknown finding_id: {finding_id}"},
+                -0.03,
+                False,
+            )
+
+        expected_type = self._scenario["finding_type"]
+        required_evidence = set(self._scenario.get("required_evidence", []))
+        supplied_evidence = set(finding.get("evidence_ids", []))
+        verified = (
+            finding.get("finding_type") == expected_type
+            and bool(required_evidence & supplied_evidence)
+        )
+        self._validated_finding_ids.add(finding_id)
+        finding["validated"] = verified
+        finding["matching_gt_id"] = self._scenario["ground_truth_id"] if verified else None
+
+        if verified and not any(
+            item["finding_id"] == finding_id for item in self._verified_findings
+        ):
+            self._verified_findings.append(dict(finding))
+
+        return (
+            {
+                "ok": True,
+                "finding_id": finding_id,
+                "verified": verified,
+                "matching_gt_id": finding["matching_gt_id"],
+            },
+            0.08 if verified else -0.02,
+            False,
+        )
+
+    def _tool_submit_report(
+        self, args: dict[str, Any]
+    ) -> tuple[dict[str, Any], float, bool]:
+        report = args.get("report_json", {})
+        score, breakdown = score_report(
+            self._scenario["task_id"],
+            report,
+            verified_findings=self._verified_findings,
+            validation_attempted=bool(self._validated_finding_ids),
+        )
+        self._last_score_breakdown = breakdown
+        return (
+            {
+                "ok": True,
+                "submitted": True,
+                "score": score,
+                "score_breakdown": breakdown,
+                "trajectory_jsonl": self.export_trajectory_jsonl(),
+            },
+            score,
+            True,
+        )

server/__init__.py

@@ -1,11 +1,11 @@
-# Copyright (c) Meta Platforms, Inc. and affiliates.
-# All rights reserved.
-#
-# This source code is licensed under the BSD-style license found in the
-# LICENSE file in the root directory of this source tree.
-
-"""Cyber Analyst environment server components."""
-
-from .Cyber_analyst_environment import CyberAnalystEnvironment
-
-__all__ = ["CyberAnalystEnvironment"]
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+"""Cyber Analyst environment server components."""
+
+from .Cyber_analyst_environment import CyberAnalystEnvironment
+
+__all__ = ["CyberAnalystEnvironment"]

server/app.py

@@ -1,79 +1,79 @@
-# Copyright (c) Meta Platforms, Inc. and affiliates.
-# All rights reserved.
-#
-# This source code is licensed under the BSD-style license found in the
-# LICENSE file in the root directory of this source tree.
-
-"""
-FastAPI application for the Cyber Analyst Environment.
-
-This module creates an HTTP server that exposes the CyberAnalystEnvironment
-over HTTP and WebSocket endpoints, compatible with EnvClient.
-
-Endpoints:
-    - POST /reset: Reset the environment
-    - POST /step: Execute an action
-    - GET /state: Get current environment state
-    - GET /schema: Get action/observation schemas
-    - WS /ws: WebSocket endpoint for persistent sessions
-
-Usage:
-    # Development (with auto-reload):
-    uvicorn server.app:app --reload --host 0.0.0.0 --port 8000
-
-    # Production:
-    uvicorn server.app:app --host 0.0.0.0 --port 8000 --workers 4
-
-    # Or run directly:
-    python -m server.app
-"""
-
-try:
-    from openenv.core.env_server.http_server import create_app
-except Exception as e:  # pragma: no cover
-    raise ImportError(
-        "openenv is required for the web interface. Install dependencies with '\n    uv sync\n'"
-    ) from e
-
-try:
-    from ..models import CyberAnalystAction, CyberAnalystObservation
-    from .Cyber_analyst_environment import CyberAnalystEnvironment
-except ImportError:
-    from models import CyberAnalystAction, CyberAnalystObservation
-    from server.Cyber_analyst_environment import CyberAnalystEnvironment
-
-
-# Create the app with web interface and README integration
-app = create_app(
-    CyberAnalystEnvironment,
-    CyberAnalystAction,
-    CyberAnalystObservation,
-    env_name="Cyber_analyst",
-    max_concurrent_envs=1,  # increase this number to allow more concurrent WebSocket sessions
-)
-
-
-def main(host: str = "0.0.0.0", port: int = 8000):
-    """
-    Entry point for direct execution via uv run or python -m.
-
-    This function enables running the server without Docker:
-        uv run --project . server
-        uv run --project . server --port 8001
-        python -m Cyber_analyst.server.app
-
-    Args:
-        host: Host address to bind to (default: "0.0.0.0")
-        port: Port number to listen on (default: 8000)
-
-    For production deployments, consider using uvicorn directly with
-    multiple workers:
-        uvicorn Cyber_analyst.server.app:app --workers 4
-    """
-    import uvicorn
-
-    uvicorn.run(app, host=host, port=port)
-
-
-if __name__ == "__main__":
-    main()
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+"""
+FastAPI application for the Cyber Analyst Environment.
+
+This module creates an HTTP server that exposes the CyberAnalystEnvironment
+over HTTP and WebSocket endpoints, compatible with EnvClient.
+
+Endpoints:
+    - POST /reset: Reset the environment
+    - POST /step: Execute an action
+    - GET /state: Get current environment state
+    - GET /schema: Get action/observation schemas
+    - WS /ws: WebSocket endpoint for persistent sessions
+
+Usage:
+    # Development (with auto-reload):
+    uvicorn server.app:app --reload --host 0.0.0.0 --port 8000
+
+    # Production:
+    uvicorn server.app:app --host 0.0.0.0 --port 8000 --workers 4
+
+    # Or run directly:
+    python -m server.app
+"""
+
+try:
+    from openenv.core.env_server.http_server import create_app
+except Exception as e:  # pragma: no cover
+    raise ImportError(
+        "openenv is required for the web interface. Install dependencies with '\n    uv sync\n'"
+    ) from e
+
+try:
+    from ..models import CyberAnalystAction, CyberAnalystObservation
+    from .Cyber_analyst_environment import CyberAnalystEnvironment
+except ImportError:
+    from models import CyberAnalystAction, CyberAnalystObservation
+    from server.Cyber_analyst_environment import CyberAnalystEnvironment
+
+
+# Create the app with web interface and README integration
+app = create_app(
+    CyberAnalystEnvironment,
+    CyberAnalystAction,
+    CyberAnalystObservation,
+    env_name="Cyber_analyst",
+    max_concurrent_envs=1,  # increase this number to allow more concurrent WebSocket sessions
+)
+
+
+def main(host: str = "0.0.0.0", port: int = 8000):
+    """
+    Entry point for direct execution via uv run or python -m.
+
+    This function enables running the server without Docker:
+        uv run --project . server
+        uv run --project . server --port 8001
+        python -m Cyber_analyst.server.app
+
+    Args:
+        host: Host address to bind to (default: "0.0.0.0")
+        port: Port number to listen on (default: 8000)
+
+    For production deployments, consider using uvicorn directly with
+    multiple workers:
+        uvicorn Cyber_analyst.server.app:app --workers 4
+    """
+    import uvicorn
+
+    uvicorn.run(app, host=host, port=port)
+
+
+if __name__ == "__main__":
+    main()

server/graders.py

@@ -1,169 +1,169 @@
-"""Deterministic graders for the Cyber Analyst OpenEnv tasks."""
-
-from __future__ import annotations
-
-import json
-from typing import Any
-
-try:
-    from .tasks import SCENARIOS
-except ImportError:  # pragma: no cover - supports direct module execution
-    from tasks import SCENARIOS
-
-
-MIN_SCORE = 0.01
-MAX_SCORE = 0.99
-
-
-def safe_reward(score: float | int | None) -> float:
-    """Clamp validator-facing scores to the strict open interval (0, 1)."""
-
-    try:
-        value = float(score if score is not None else 0.0)
-    except (TypeError, ValueError):
-        value = 0.0
-    return max(MIN_SCORE, min(MAX_SCORE, value))
-
-
-def _coerce_report(report: Any) -> dict[str, Any]:
-    if isinstance(report, dict):
-        return report
-    if isinstance(report, str):
-        try:
-            decoded = json.loads(report)
-        except json.JSONDecodeError:
-            return {"summary": report, "findings": []}
-        return decoded if isinstance(decoded, dict) else {"findings": []}
-    return {"findings": []}
-
-
-def _text_contains_any(text: str, keywords: list[str]) -> bool:
-    lowered = text.lower()
-    return any(keyword.lower() in lowered for keyword in keywords)
-
-
-def _report_findings(report: dict[str, Any]) -> list[dict[str, Any]]:
-    findings = report.get("findings", [])
-    if isinstance(findings, dict):
-        findings = [findings]
-    return [finding for finding in findings if isinstance(finding, dict)]
-
-
-def score_report(
-    task_id: str,
-    report: Any,
-    verified_findings: list[dict[str, Any]] | None = None,
-    validation_attempted: bool = False,
-) -> tuple[float, dict[str, Any]]:
-    """Score a submitted report against one task's deterministic ground truth."""
-
-    scenario = SCENARIOS.get(task_id)
-    report_dict = _coerce_report(report)
-    report_findings = _report_findings(report_dict)
-    verified_findings = verified_findings or []
-
-    if scenario is None:
-        return MIN_SCORE, {"unknown_task": task_id}
-
-    expected_type = scenario["finding_type"]
-    expected_evidence = set(scenario.get("required_evidence", [])) | set(
-        scenario.get("supporting_evidence", [])
-    )
-
-    matching_verified = [
-        finding
-        for finding in verified_findings
-        if finding.get("finding_type") == expected_type
-    ]
-    matching_report = [
-        finding for finding in report_findings if finding.get("finding_type") == expected_type
-    ]
-
-    score = 0.05
-    breakdown: dict[str, Any] = {
-        "base": 0.05,
-        "verified_correct": 0.0,
-        "valid_evidence": 0.0,
-        "actionable_remediation": 0.0,
-        "hallucination_penalty": 0.0,
-        "validation_penalty": 0.0,
-    }
-
-    if matching_verified and matching_report:
-        impact_text = " ".join(
-            str(finding.get("impact", "")) + " " + str(finding.get("description", ""))
-            for finding in matching_report
-        )
-        if _text_contains_any(impact_text, scenario.get("impact_keywords", [])):
-            score += 0.60
-            breakdown["verified_correct"] = 0.60
-
-    report_evidence: set[str] = set()
-    for finding in matching_report:
-        evidence_ids = finding.get("evidence_ids", [])
-        if isinstance(evidence_ids, str):
-            evidence_ids = [evidence_ids]
-        report_evidence.update(str(evidence_id) for evidence_id in evidence_ids)
-
-    if report_evidence & expected_evidence:
-        score += 0.15
-        breakdown["valid_evidence"] = 0.15
-
-    remediation_text = " ".join(
-        str(finding.get("remediation", "")) for finding in matching_report
-    )
-    if _text_contains_any(remediation_text, scenario.get("remediation_keywords", [])):
-        score += 0.15
-        breakdown["actionable_remediation"] = 0.15
-
-    verified_types = {finding.get("finding_type") for finding in verified_findings}
-    hallucinated = [
-        finding
-        for finding in report_findings
-        if finding.get("finding_type") not in verified_types
-    ]
-    if hallucinated:
-        penalty = 0.40 * len(hallucinated)
-        score -= penalty
-        breakdown["hallucination_penalty"] = -penalty
-
-    if not validation_attempted:
-        score -= 0.20
-        breakdown["validation_penalty"] = -0.20
-
-    final_score = safe_reward(score)
-    breakdown["raw_score"] = round(score, 4)
-    breakdown["score"] = final_score
-    return final_score, breakdown
-
-
-def _payload_from_args(*args: Any, **kwargs: Any) -> dict[str, Any]:
-    if args and isinstance(args[0], dict):
-        payload = dict(args[0])
-    else:
-        payload = {}
-    payload.update(kwargs)
-    return payload
-
-
-def grade_task(task_id: str, *args: Any, **kwargs: Any) -> float:
-    """Manifest-friendly grader adapter."""
-
-    payload = _payload_from_args(*args, **kwargs)
-    report = payload.get("report") or payload.get("report_json") or payload
-    verified_findings = payload.get("verified_findings", [])
-    validation_attempted = bool(payload.get("validation_attempted", False))
-    score, _ = score_report(task_id, report, verified_findings, validation_attempted)
-    return score
-
-
-def grade_secret_exposure_easy(*args: Any, **kwargs: Any) -> float:
-    return grade_task("secret_exposure_easy", *args, **kwargs)
-
-
-def grade_missing_security_headers_medium(*args: Any, **kwargs: Any) -> float:
-    return grade_task("missing_security_headers_medium", *args, **kwargs)
-
-
-def grade_authz_boundary_hard(*args: Any, **kwargs: Any) -> float:
-    return grade_task("authz_boundary_hard", *args, **kwargs)
+"""Deterministic graders for the Cyber Analyst OpenEnv tasks."""
+
+from __future__ import annotations
+
+import json
+from typing import Any
+
+try:
+    from .tasks import SCENARIOS
+except ImportError:  # pragma: no cover - supports direct module execution
+    from tasks import SCENARIOS
+
+
+MIN_SCORE = 0.01
+MAX_SCORE = 0.99
+
+
+def safe_reward(score: float | int | None) -> float:
+    """Clamp validator-facing scores to the strict open interval (0, 1)."""
+
+    try:
+        value = float(score if score is not None else 0.0)
+    except (TypeError, ValueError):
+        value = 0.0
+    return max(MIN_SCORE, min(MAX_SCORE, value))
+
+
+def _coerce_report(report: Any) -> dict[str, Any]:
+    if isinstance(report, dict):
+        return report
+    if isinstance(report, str):
+        try:
+            decoded = json.loads(report)
+        except json.JSONDecodeError:
+            return {"summary": report, "findings": []}
+        return decoded if isinstance(decoded, dict) else {"findings": []}
+    return {"findings": []}
+
+
+def _text_contains_any(text: str, keywords: list[str]) -> bool:
+    lowered = text.lower()
+    return any(keyword.lower() in lowered for keyword in keywords)
+
+
+def _report_findings(report: dict[str, Any]) -> list[dict[str, Any]]:
+    findings = report.get("findings", [])
+    if isinstance(findings, dict):
+        findings = [findings]
+    return [finding for finding in findings if isinstance(finding, dict)]
+
+
+def score_report(
+    task_id: str,
+    report: Any,
+    verified_findings: list[dict[str, Any]] | None = None,
+    validation_attempted: bool = False,
+) -> tuple[float, dict[str, Any]]:
+    """Score a submitted report against one task's deterministic ground truth."""
+
+    scenario = SCENARIOS.get(task_id)
+    report_dict = _coerce_report(report)
+    report_findings = _report_findings(report_dict)
+    verified_findings = verified_findings or []
+
+    if scenario is None:
+        return MIN_SCORE, {"unknown_task": task_id}
+
+    expected_type = scenario["finding_type"]
+    expected_evidence = set(scenario.get("required_evidence", [])) | set(
+        scenario.get("supporting_evidence", [])
+    )
+
+    matching_verified = [
+        finding
+        for finding in verified_findings
+        if finding.get("finding_type") == expected_type
+    ]
+    matching_report = [
+        finding for finding in report_findings if finding.get("finding_type") == expected_type
+    ]
+
+    score = 0.05
+    breakdown: dict[str, Any] = {
+        "base": 0.05,
+        "verified_correct": 0.0,
+        "valid_evidence": 0.0,
+        "actionable_remediation": 0.0,
+        "hallucination_penalty": 0.0,
+        "validation_penalty": 0.0,
+    }
+
+    if matching_verified and matching_report:
+        impact_text = " ".join(
+            str(finding.get("impact", "")) + " " + str(finding.get("description", ""))
+            for finding in matching_report
+        )
+        if _text_contains_any(impact_text, scenario.get("impact_keywords", [])):
+            score += 0.60
+            breakdown["verified_correct"] = 0.60
+
+    report_evidence: set[str] = set()
+    for finding in matching_report:
+        evidence_ids = finding.get("evidence_ids", [])
+        if isinstance(evidence_ids, str):
+            evidence_ids = [evidence_ids]
+        report_evidence.update(str(evidence_id) for evidence_id in evidence_ids)
+
+    if report_evidence & expected_evidence:
+        score += 0.15
+        breakdown["valid_evidence"] = 0.15
+
+    remediation_text = " ".join(
+        str(finding.get("remediation", "")) for finding in matching_report
+    )
+    if _text_contains_any(remediation_text, scenario.get("remediation_keywords", [])):
+        score += 0.15
+        breakdown["actionable_remediation"] = 0.15
+
+    verified_types = {finding.get("finding_type") for finding in verified_findings}
+    hallucinated = [
+        finding
+        for finding in report_findings
+        if finding.get("finding_type") not in verified_types
+    ]
+    if hallucinated:
+        penalty = 0.40 * len(hallucinated)
+        score -= penalty
+        breakdown["hallucination_penalty"] = -penalty
+
+    if not validation_attempted:
+        score -= 0.20
+        breakdown["validation_penalty"] = -0.20
+
+    final_score = safe_reward(score)
+    breakdown["raw_score"] = round(score, 4)
+    breakdown["score"] = final_score
+    return final_score, breakdown
+
+
+def _payload_from_args(*args: Any, **kwargs: Any) -> dict[str, Any]:
+    if args and isinstance(args[0], dict):
+        payload = dict(args[0])
+    else:
+        payload = {}
+    payload.update(kwargs)
+    return payload
+
+
+def grade_task(task_id: str, *args: Any, **kwargs: Any) -> float:
+    """Manifest-friendly grader adapter."""
+
+    payload = _payload_from_args(*args, **kwargs)
+    report = payload.get("report") or payload.get("report_json") or payload
+    verified_findings = payload.get("verified_findings", [])
+    validation_attempted = bool(payload.get("validation_attempted", False))
+    score, _ = score_report(task_id, report, verified_findings, validation_attempted)
+    return score
+
+
+def grade_secret_exposure_easy(*args: Any, **kwargs: Any) -> float:
+    return grade_task("secret_exposure_easy", *args, **kwargs)
+
+
+def grade_missing_security_headers_medium(*args: Any, **kwargs: Any) -> float:
+    return grade_task("missing_security_headers_medium", *args, **kwargs)
+
+
+def grade_authz_boundary_hard(*args: Any, **kwargs: Any) -> float:
+    return grade_task("authz_boundary_hard", *args, **kwargs)

server/requirements.txt

@@ -1,6 +1,6 @@
-openenv[core]>=0.2.0
-fastapi>=0.115.0
-uvicorn>=0.24.0
-openai>=1.0.0
-
-
+openenv[core]>=0.2.0
+fastapi>=0.115.0
+uvicorn>=0.24.0
+openai>=1.0.0
+
+

server/tasks.py

@@ -1,283 +1,283 @@
-"""Deterministic scenario registry for SecOps Evidence Gym."""
-
-from __future__ import annotations
-
-from copy import deepcopy
-from random import Random
-from typing import Any
-
-
-DEFAULT_TASK_ID = "secret_exposure_easy"
-
-TOOL_CATALOG: list[dict[str, Any]] = [
-    {
-        "name": "list_assets",
-        "description": "List synthetic services, routes, and artifact collections.",
-        "args": {},
-    },
-    {
-        "name": "get_log_events",
-        "description": "Return sanitized telemetry evidence ids for a service/query.",
-        "args": {"service_id": "str", "query": "str"},
-    },
-    {
-        "name": "check_security_headers",
-        "description": "Inspect a service header snapshot and return pass/fail evidence.",
-        "args": {"service_id": "str"},
-    },
-    {
-        "name": "search_repo",
-        "description": "Search synthetic repo/config snippets for evidence ids.",
-        "args": {"query": "str"},
-    },
-    {
-        "name": "scan_dependencies",
-        "description": "Inspect a synthetic dependency manifest excerpt.",
-        "args": {},
-    },
-    {
-        "name": "create_finding",
-        "description": "Store a candidate finding for verifier review.",
-        "args": {
-            "finding_type": "str",
-            "evidence_ids": "list[str]",
-            "severity_guess": "str",
-            "remediation": "str",
-        },
-    },
-    {
-        "name": "validate_finding",
-        "description": "Run the deterministic verifier for a candidate finding.",
-        "args": {"finding_id": "str"},
-    },
-    {
-        "name": "submit_report",
-        "description": "Submit the final structured report and end the episode.",
-        "args": {"report_json": "dict"},
-    },
-]
-
-
-SCENARIOS: dict[str, dict[str, Any]] = {
-    "secret_exposure_easy": {
-        "task_id": "secret_exposure_easy",
-        "difficulty": "easy",
-        "title": "Secret exposure in repo snapshot",
-        "alert": (
-            "Repository hygiene alert: a synthetic config snapshot may contain "
-            "credential-like material. Investigate, cite evidence, validate, and "
-            "submit a remediation report."
-        ),
-        "ground_truth_id": "GT-SECRET-001",
-        "finding_type": "secret_exposure",
-        "severity": "high",
-        "required_evidence": ["EVID-101"],
-        "remediation_keywords": ["rotate", "remove"],
-        "impact_keywords": ["secret", "credential", "api key", "leak"],
-        "assets": {
-            "services": ["gateway", "profile-service", "admin-service"],
-            "routes": ["/login", "/profile", "/admin/export"],
-            "collections": ["repo_snapshot", "telemetry", "headers", "dependencies"],
-        },
-        "repo": [
-            {
-                "evidence_id": "EVID-101",
-                "source": "repo_snapshot/config/profile-service.env",
-                "service_id": "profile-service",
-                "tags": ["api key", "secret", "credential", "config"],
-                "text": (
-                    "PROFILE_EXPORT_API_KEY=CYBERLAB_DO_NOT_USE_API_KEY_7F3A "
-                    "# synthetic lab credential"
-                ),
-            },
-            {
-                "evidence_id": "EVID-102",
-                "source": "repo_snapshot/config/gateway.env",
-                "service_id": "gateway",
-                "tags": ["config", "benign"],
-                "text": "LOG_LEVEL=info; FEATURE_SAFE_HEADERS=true",
-            },
-        ],
-        "logs": [
-            {
-                "evidence_id": "EVID-103",
-                "service_id": "profile-service",
-                "tags": ["benign", "startup"],
-                "text": "profile-service boot completed for synthetic tenant secops-lab",
-            }
-        ],
-        "headers": {
-            "gateway": {
-                "evidence_id": "EVID-104",
-                "present": ["Strict-Transport-Security", "Content-Security-Policy"],
-                "missing": [],
-            }
-        },
-        "dependencies": [
-            {
-                "evidence_id": "EVID-105",
-                "source": "repo_snapshot/requirements.lock",
-                "tags": ["dependency", "benign"],
-                "text": "fastapi==0.115.0; pydantic==2.8.2",
-            }
-        ],
-    },
-    "missing_security_headers_medium": {
-        "task_id": "missing_security_headers_medium",
-        "difficulty": "medium",
-        "title": "Missing security headers",
-        "alert": (
-            "Gateway response-hardening alert: verify whether required security "
-            "headers are missing or weak and submit evidence-backed remediation."
-        ),
-        "ground_truth_id": "GT-HEADERS-001",
-        "finding_type": "missing_security_headers",
-        "severity": "medium",
-        "required_evidence": ["EVID-201"],
-        "remediation_keywords": ["hsts", "csp"],
-        "impact_keywords": ["header", "hsts", "csp", "clickjacking"],
-        "assets": {
-            "services": ["gateway", "profile-service", "admin-service"],
-            "routes": ["/login", "/profile", "/admin/export"],
-            "collections": ["repo_snapshot", "telemetry", "headers", "dependencies"],
-        },
-        "repo": [
-            {
-                "evidence_id": "EVID-202",
-                "source": "repo_snapshot/gateway/security_headers.py",
-                "service_id": "gateway",
-                "tags": ["headers", "config"],
-                "text": "X-Frame-Options is set, but HSTS and CSP are not configured.",
-            }
-        ],
-        "logs": [
-            {
-                "evidence_id": "EVID-203",
-                "service_id": "gateway",
-                "tags": ["benign", "response"],
-                "text": "GET /profile 200 request_id=req-442 synthetic header audit",
-            }
-        ],
-        "headers": {
-            "gateway": {
-                "evidence_id": "EVID-201",
-                "present": ["X-Frame-Options", "X-Content-Type-Options"],
-                "missing": ["Strict-Transport-Security", "Content-Security-Policy"],
-            }
-        },
-        "dependencies": [
-            {
-                "evidence_id": "EVID-204",
-                "source": "repo_snapshot/requirements.lock",
-                "tags": ["dependency", "benign"],
-                "text": "starlette==0.38.2; uvicorn==0.30.1",
-            }
-        ],
-    },
-    "authz_boundary_hard": {
-        "task_id": "authz_boundary_hard",
-        "difficulty": "hard",
-        "title": "Authorisation boundary misconfiguration",
-        "alert": (
-            "Access-control drift alert: investigate a route/role mismatch in the "
-            "synthetic admin service and submit a validated remediation report."
-        ),
-        "ground_truth_id": "GT-AUTHZ-001",
-        "finding_type": "authz_boundary_misconfiguration",
-        "severity": "critical",
-        "required_evidence": ["EVID-301"],
-        "supporting_evidence": ["EVID-302"],
-        "remediation_keywords": ["least privilege", "policy", "regression"],
-        "impact_keywords": ["authorization", "authorisation", "role", "admin"],
-        "assets": {
-            "services": ["gateway", "profile-service", "admin-service"],
-            "routes": ["/login", "/profile", "/admin/export"],
-            "collections": ["repo_snapshot", "telemetry", "headers", "dependencies"],
-        },
-        "repo": [
-            {
-                "evidence_id": "EVID-301",
-                "source": "repo_snapshot/admin-service/policy_matrix.yaml",
-                "service_id": "admin-service",
-                "tags": ["authorization", "role", "policy", "admin export"],
-                "text": (
-                    "route=/admin/export allowed_roles=[admin, analyst] "
-                    "expected_roles=[admin]"
-                ),
-            }
-        ],
-        "logs": [
-            {
-                "evidence_id": "EVID-302",
-                "service_id": "admin-service",
-                "tags": ["authorization", "role", "admin export"],
-                "text": (
-                    "request_id=req-913 route=/admin/export role=analyst "
-                    "decision=allow synthetic boundary-check event"
-                ),
-            },
-            {
-                "evidence_id": "EVID-303",
-                "service_id": "gateway",
-                "tags": ["benign", "auth"],
-                "text": "request_id=req-912 route=/profile role=user decision=allow",
-            },
-        ],
-        "headers": {
-            "admin-service": {
-                "evidence_id": "EVID-304",
-                "present": ["Strict-Transport-Security", "Content-Security-Policy"],
-                "missing": [],
-            }
-        },
-        "dependencies": [
-            {
-                "evidence_id": "EVID-305",
-                "source": "repo_snapshot/requirements.lock",
-                "tags": ["dependency", "benign"],
-                "text": "pyyaml==6.0.2; fastapi==0.115.0",
-            }
-        ],
-    },
-}
-
-
-def list_task_ids() -> list[str]:
-    return list(SCENARIOS)
-
-
-def build_scenario(task_id: str | None, seed: int | None = None) -> dict[str, Any]:
-    """Return a deep-copied scenario with deterministic benign variation."""
-
-    selected_task_id = task_id if task_id in SCENARIOS else DEFAULT_TASK_ID
-    scenario = deepcopy(SCENARIOS[selected_task_id])
-    scenario["seed"] = seed
-
-    rng = Random(seed if seed is not None else 0)
-    service_alias_sets = [
-        ["gateway", "profile-service", "admin-service"],
-        ["edge-gateway", "user-profile", "admin-service"],
-        ["public-gateway", "profile-api", "backoffice-admin"],
-    ]
-    aliases = service_alias_sets[rng.randrange(len(service_alias_sets))]
-    original_services = scenario["assets"]["services"]
-    alias_map = dict(zip(original_services, aliases, strict=True))
-
-    scenario["service_aliases"] = alias_map
-    scenario["assets"]["services"] = [alias_map.get(s, s) for s in original_services]
-
-    for collection_name in ("repo", "logs"):
-        for item in scenario.get(collection_name, []):
-            service_id = item.get("service_id")
-            if service_id in alias_map:
-                item["service_id"] = alias_map[service_id]
-
-    scenario["headers"] = {
-        alias_map.get(service_id, service_id): snapshot
-        for service_id, snapshot in scenario.get("headers", {}).items()
-    }
-
-    for entries_name in ("repo", "logs", "dependencies"):
-        rng.shuffle(scenario.get(entries_name, []))
-
-    return scenario
+"""Deterministic scenario registry for SecOps Evidence Gym."""
+
+from __future__ import annotations
+
+from copy import deepcopy
+from random import Random
+from typing import Any
+
+
+DEFAULT_TASK_ID = "secret_exposure_easy"
+
+TOOL_CATALOG: list[dict[str, Any]] = [
+    {
+        "name": "list_assets",
+        "description": "List synthetic services, routes, and artifact collections.",
+        "args": {},
+    },
+    {
+        "name": "get_log_events",
+        "description": "Return sanitized telemetry evidence ids for a service/query.",
+        "args": {"service_id": "str", "query": "str"},
+    },
+    {
+        "name": "check_security_headers",
+        "description": "Inspect a service header snapshot and return pass/fail evidence.",
+        "args": {"service_id": "str"},
+    },
+    {
+        "name": "search_repo",
+        "description": "Search synthetic repo/config snippets for evidence ids.",
+        "args": {"query": "str"},
+    },
+    {
+        "name": "scan_dependencies",
+        "description": "Inspect a synthetic dependency manifest excerpt.",
+        "args": {},
+    },
+    {
+        "name": "create_finding",
+        "description": "Store a candidate finding for verifier review.",
+        "args": {
+            "finding_type": "str",
+            "evidence_ids": "list[str]",
+            "severity_guess": "str",
+            "remediation": "str",
+        },
+    },
+    {
+        "name": "validate_finding",
+        "description": "Run the deterministic verifier for a candidate finding.",
+        "args": {"finding_id": "str"},
+    },
+    {
+        "name": "submit_report",
+        "description": "Submit the final structured report and end the episode.",
+        "args": {"report_json": "dict"},
+    },
+]
+
+
+SCENARIOS: dict[str, dict[str, Any]] = {
+    "secret_exposure_easy": {
+        "task_id": "secret_exposure_easy",
+        "difficulty": "easy",
+        "title": "Secret exposure in repo snapshot",
+        "alert": (
+            "Repository hygiene alert: a synthetic config snapshot may contain "
+            "credential-like material. Investigate, cite evidence, validate, and "
+            "submit a remediation report."
+        ),
+        "ground_truth_id": "GT-SECRET-001",
+        "finding_type": "secret_exposure",
+        "severity": "high",
+        "required_evidence": ["EVID-101"],
+        "remediation_keywords": ["rotate", "remove"],
+        "impact_keywords": ["secret", "credential", "api key", "leak"],
+        "assets": {
+            "services": ["gateway", "profile-service", "admin-service"],
+            "routes": ["/login", "/profile", "/admin/export"],
+            "collections": ["repo_snapshot", "telemetry", "headers", "dependencies"],
+        },
+        "repo": [
+            {
+                "evidence_id": "EVID-101",
+                "source": "repo_snapshot/config/profile-service.env",
+                "service_id": "profile-service",
+                "tags": ["api key", "secret", "credential", "config"],
+                "text": (
+                    "PROFILE_EXPORT_API_KEY=CYBERLAB_DO_NOT_USE_API_KEY_7F3A "
+                    "# synthetic lab credential"
+                ),
+            },
+            {
+                "evidence_id": "EVID-102",
+                "source": "repo_snapshot/config/gateway.env",
+                "service_id": "gateway",
+                "tags": ["config", "benign"],
+                "text": "LOG_LEVEL=info; FEATURE_SAFE_HEADERS=true",
+            },
+        ],
+        "logs": [
+            {
+                "evidence_id": "EVID-103",
+                "service_id": "profile-service",
+                "tags": ["benign", "startup"],
+                "text": "profile-service boot completed for synthetic tenant secops-lab",
+            }
+        ],
+        "headers": {
+            "gateway": {
+                "evidence_id": "EVID-104",
+                "present": ["Strict-Transport-Security", "Content-Security-Policy"],
+                "missing": [],
+            }
+        },
+        "dependencies": [
+            {
+                "evidence_id": "EVID-105",
+                "source": "repo_snapshot/requirements.lock",
+                "tags": ["dependency", "benign"],
+                "text": "fastapi==0.115.0; pydantic==2.8.2",
+            }
+        ],
+    },
+    "missing_security_headers_medium": {
+        "task_id": "missing_security_headers_medium",
+        "difficulty": "medium",
+        "title": "Missing security headers",
+        "alert": (
+            "Gateway response-hardening alert: verify whether required security "
+            "headers are missing or weak and submit evidence-backed remediation."
+        ),
+        "ground_truth_id": "GT-HEADERS-001",
+        "finding_type": "missing_security_headers",
+        "severity": "medium",
+        "required_evidence": ["EVID-201"],
+        "remediation_keywords": ["hsts", "csp"],
+        "impact_keywords": ["header", "hsts", "csp", "clickjacking"],
+        "assets": {
+            "services": ["gateway", "profile-service", "admin-service"],
+            "routes": ["/login", "/profile", "/admin/export"],
+            "collections": ["repo_snapshot", "telemetry", "headers", "dependencies"],
+        },
+        "repo": [
+            {
+                "evidence_id": "EVID-202",
+                "source": "repo_snapshot/gateway/security_headers.py",
+                "service_id": "gateway",
+                "tags": ["headers", "config"],
+                "text": "X-Frame-Options is set, but HSTS and CSP are not configured.",
+            }
+        ],
+        "logs": [
+            {
+                "evidence_id": "EVID-203",
+                "service_id": "gateway",
+                "tags": ["benign", "response"],
+                "text": "GET /profile 200 request_id=req-442 synthetic header audit",
+            }
+        ],
+        "headers": {
+            "gateway": {
+                "evidence_id": "EVID-201",
+                "present": ["X-Frame-Options", "X-Content-Type-Options"],
+                "missing": ["Strict-Transport-Security", "Content-Security-Policy"],
+            }
+        },
+        "dependencies": [
+            {
+                "evidence_id": "EVID-204",
+                "source": "repo_snapshot/requirements.lock",
+                "tags": ["dependency", "benign"],
+                "text": "starlette==0.38.2; uvicorn==0.30.1",
+            }
+        ],
+    },
+    "authz_boundary_hard": {
+        "task_id": "authz_boundary_hard",
+        "difficulty": "hard",
+        "title": "Authorisation boundary misconfiguration",
+        "alert": (
+            "Access-control drift alert: investigate a route/role mismatch in the "
+            "synthetic admin service and submit a validated remediation report."
+        ),
+        "ground_truth_id": "GT-AUTHZ-001",
+        "finding_type": "authz_boundary_misconfiguration",
+        "severity": "critical",
+        "required_evidence": ["EVID-301"],
+        "supporting_evidence": ["EVID-302"],
+        "remediation_keywords": ["least privilege", "policy", "regression"],
+        "impact_keywords": ["authorization", "authorisation", "role", "admin"],
+        "assets": {
+            "services": ["gateway", "profile-service", "admin-service"],
+            "routes": ["/login", "/profile", "/admin/export"],
+            "collections": ["repo_snapshot", "telemetry", "headers", "dependencies"],
+        },
+        "repo": [
+            {
+                "evidence_id": "EVID-301",
+                "source": "repo_snapshot/admin-service/policy_matrix.yaml",
+                "service_id": "admin-service",
+                "tags": ["authorization", "role", "policy", "admin export"],
+                "text": (
+                    "route=/admin/export allowed_roles=[admin, analyst] "
+                    "expected_roles=[admin]"
+                ),
+            }
+        ],
+        "logs": [
+            {
+                "evidence_id": "EVID-302",
+                "service_id": "admin-service",
+                "tags": ["authorization", "role", "admin export"],
+                "text": (
+                    "request_id=req-913 route=/admin/export role=analyst "
+                    "decision=allow synthetic boundary-check event"
+                ),
+            },
+            {
+                "evidence_id": "EVID-303",
+                "service_id": "gateway",
+                "tags": ["benign", "auth"],
+                "text": "request_id=req-912 route=/profile role=user decision=allow",
+            },
+        ],
+        "headers": {
+            "admin-service": {
+                "evidence_id": "EVID-304",
+                "present": ["Strict-Transport-Security", "Content-Security-Policy"],
+                "missing": [],
+            }
+        },
+        "dependencies": [
+            {
+                "evidence_id": "EVID-305",
+                "source": "repo_snapshot/requirements.lock",
+                "tags": ["dependency", "benign"],
+                "text": "pyyaml==6.0.2; fastapi==0.115.0",
+            }
+        ],
+    },
+}
+
+
+def list_task_ids() -> list[str]:
+    return list(SCENARIOS)
+
+
+def build_scenario(task_id: str | None, seed: int | None = None) -> dict[str, Any]:
+    """Return a deep-copied scenario with deterministic benign variation."""
+
+    selected_task_id = task_id if task_id in SCENARIOS else DEFAULT_TASK_ID
+    scenario = deepcopy(SCENARIOS[selected_task_id])
+    scenario["seed"] = seed
+
+    rng = Random(seed if seed is not None else 0)
+    service_alias_sets = [
+        ["gateway", "profile-service", "admin-service"],
+        ["edge-gateway", "user-profile", "admin-service"],
+        ["public-gateway", "profile-api", "backoffice-admin"],
+    ]
+    aliases = service_alias_sets[rng.randrange(len(service_alias_sets))]
+    original_services = scenario["assets"]["services"]
+    alias_map = dict(zip(original_services, aliases, strict=True))
+
+    scenario["service_aliases"] = alias_map
+    scenario["assets"]["services"] = [alias_map.get(s, s) for s in original_services]
+
+    for collection_name in ("repo", "logs"):
+        for item in scenario.get(collection_name, []):
+            service_id = item.get("service_id")
+            if service_id in alias_map:
+                item["service_id"] = alias_map[service_id]
+
+    scenario["headers"] = {
+        alias_map.get(service_id, service_id): snapshot
+        for service_id, snapshot in scenario.get("headers", {}).items()
+    }
+
+    for entries_name in ("repo", "logs", "dependencies"):
+        rng.shuffle(scenario.get(entries_name, []))
+
+    return scenario

tests/conftest.py

@@ -1,7 +1,7 @@
-import sys
-from pathlib import Path
-
-
-PACKAGE_PARENT = Path(__file__).resolve().parents[2]
-if str(PACKAGE_PARENT) not in sys.path:
-    sys.path.insert(0, str(PACKAGE_PARENT))
+import sys
+from pathlib import Path
+
+
+PACKAGE_PARENT = Path(__file__).resolve().parents[2]
+if str(PACKAGE_PARENT) not in sys.path:
+    sys.path.insert(0, str(PACKAGE_PARENT))

tests/test_environment.py

@@ -1,187 +1,187 @@
-from Cyber_analyst.models import CyberAnalystAction
-from Cyber_analyst.server.Cyber_analyst_environment import CyberAnalystEnvironment
-from Cyber_analyst.server.graders import (
-    grade_authz_boundary_hard,
-    grade_missing_security_headers_medium,
-    grade_secret_exposure_easy,
-    safe_reward,
-)
-
-
-def _run_success_path(task_id, actions):
-    env = CyberAnalystEnvironment()
-    obs = env.reset(task_id=task_id, seed=7)
-    assert obs.task_id == task_id
-
-    for action in actions:
-        obs = env.step(action)
-
-    assert obs.done is True
-    assert obs.tool_result["score"] > 0.5
-    assert 0.01 <= obs.tool_result["score"] <= 0.99
-    assert obs.error == ""
-    return obs
-
-
-def test_secret_exposure_success_path():
-    report = {
-        "findings": [
-            {
-                "finding_type": "secret_exposure",
-                "evidence_ids": ["EVID-101"],
-                "impact": "A synthetic API key secret is exposed in config.",
-                "remediation": "Remove the key and rotate the credential.",
-            }
-        ]
-    }
-    obs = _run_success_path(
-        "secret_exposure_easy",
-        [
-            CyberAnalystAction(tool_name="search_repo", args={"query": "api key"}),
-            CyberAnalystAction(
-                tool_name="create_finding",
-                args={
-                    "finding_type": "secret_exposure",
-                    "evidence_ids": ["EVID-101"],
-                    "severity_guess": "high",
-                    "remediation": "Remove and rotate the synthetic credential.",
-                },
-            ),
-            CyberAnalystAction(tool_name="validate_finding", args={"finding_id": "FND-001"}),
-            CyberAnalystAction(tool_name="submit_report", args={"report_json": report}),
-        ],
-    )
-    assert obs.verified_findings[0]["matching_gt_id"] == "GT-SECRET-001"
-    assert "trajectory_jsonl" in obs.tool_result
-    assert "search_repo" in obs.tool_result["trajectory_jsonl"]
-
-
-def test_missing_security_headers_success_path():
-    report = {
-        "findings": [
-            {
-                "finding_type": "missing_security_headers",
-                "evidence_ids": ["EVID-201"],
-                "impact": "The gateway is missing HSTS and CSP headers.",
-                "remediation": "Add HSTS and CSP at the gateway.",
-            }
-        ]
-    }
-    obs = _run_success_path(
-        "missing_security_headers_medium",
-        [
-            CyberAnalystAction(
-                tool_name="check_security_headers", args={"service_id": "gateway"}
-            ),
-            CyberAnalystAction(
-                tool_name="create_finding",
-                args={
-                    "finding_type": "missing_security_headers",
-                    "evidence_ids": ["EVID-201"],
-                    "severity_guess": "medium",
-                    "remediation": "Add HSTS and CSP headers.",
-                },
-            ),
-            CyberAnalystAction(tool_name="validate_finding", args={"finding_id": "FND-001"}),
-            CyberAnalystAction(tool_name="submit_report", args={"report_json": report}),
-        ],
-    )
-    assert obs.score_breakdown["valid_evidence"] == 0.15
-
-
-def test_authz_boundary_success_path_with_alias_compatible_service_ids():
-    report = {
-        "findings": [
-            {
-                "finding_type": "authz_boundary_misconfiguration",
-                "evidence_ids": ["EVID-301", "EVID-302"],
-                "impact": "The admin route authorization policy allows an analyst role.",
-                "remediation": "Apply least privilege in the policy and add a regression test.",
-            }
-        ]
-    }
-    obs = _run_success_path(
-        "authz_boundary_hard",
-        [
-            CyberAnalystAction(tool_name="list_assets", args={}),
-            CyberAnalystAction(
-                tool_name="get_log_events",
-                args={"service_id": "admin-service", "query": "admin export"},
-            ),
-            CyberAnalystAction(tool_name="search_repo", args={"query": "admin export"}),
-            CyberAnalystAction(
-                tool_name="create_finding",
-                args={
-                    "finding_type": "authz_boundary_misconfiguration",
-                    "evidence_ids": ["EVID-301", "EVID-302"],
-                    "severity_guess": "critical",
-                    "remediation": "Apply least privilege and add a regression test.",
-                },
-            ),
-            CyberAnalystAction(tool_name="validate_finding", args={"finding_id": "FND-001"}),
-            CyberAnalystAction(tool_name="submit_report", args={"report_json": report}),
-        ],
-    )
-    assert obs.score_breakdown["actionable_remediation"] == 0.15
-
-
-def test_invalid_tool_returns_observation_error():
-    env = CyberAnalystEnvironment()
-    env.reset(task_id="secret_exposure_easy", seed=1)
-    obs = env.step(CyberAnalystAction(tool_name="shell", args={"cmd": "whoami"}))
-    assert obs.done is False
-    assert obs.error == "unsupported_tool"
-    assert obs.tool_result["ok"] is False
-
-
-def test_hallucinated_report_scores_low_but_in_range():
-    env = CyberAnalystEnvironment()
-    env.reset(task_id="secret_exposure_easy", seed=1)
-    obs = env.step(
-        CyberAnalystAction(
-            tool_name="submit_report",
-            args={
-                "report_json": {
-                    "findings": [
-                        {
-                            "finding_type": "remote_code_execution",
-                            "evidence_ids": [],
-                            "impact": "Unsupported claim.",
-                            "remediation": "Unsupported remediation.",
-                        }
-                    ]
-                }
-            },
-        )
-    )
-    assert obs.done is True
-    assert obs.tool_result["score"] == 0.01
-
-
-def test_repeated_action_hard_stops_episode():
-    env = CyberAnalystEnvironment()
-    env.reset(task_id="secret_exposure_easy", seed=1)
-    obs = None
-    for _ in range(6):
-        obs = env.step(CyberAnalystAction(tool_name="list_assets", args={}))
-    assert obs is not None
-    assert obs.done is True
-    assert obs.error == "repeat_hard_stop"
-
-
-def test_seed_determinism_for_assets():
-    env_one = CyberAnalystEnvironment()
-    env_two = CyberAnalystEnvironment()
-    env_one.reset(task_id="authz_boundary_hard", seed=22)
-    env_two.reset(task_id="authz_boundary_hard", seed=22)
-    obs_one = env_one.step(CyberAnalystAction(tool_name="list_assets", args={}))
-    obs_two = env_two.step(CyberAnalystAction(tool_name="list_assets", args={}))
-    assert obs_one.tool_result == obs_two.tool_result
-
-
-def test_grader_adapters_and_clamp_are_strictly_in_range():
-    assert safe_reward(-1) == 0.01
-    assert safe_reward(2) == 0.99
-    assert 0.01 <= grade_secret_exposure_easy() <= 0.99
-    assert 0.01 <= grade_missing_security_headers_medium() <= 0.99
-    assert 0.01 <= grade_authz_boundary_hard() <= 0.99
+from Cyber_analyst.models import CyberAnalystAction
+from Cyber_analyst.server.Cyber_analyst_environment import CyberAnalystEnvironment
+from Cyber_analyst.server.graders import (
+    grade_authz_boundary_hard,
+    grade_missing_security_headers_medium,
+    grade_secret_exposure_easy,
+    safe_reward,
+)
+
+
+def _run_success_path(task_id, actions):
+    env = CyberAnalystEnvironment()
+    obs = env.reset(task_id=task_id, seed=7)
+    assert obs.task_id == task_id
+
+    for action in actions:
+        obs = env.step(action)
+
+    assert obs.done is True
+    assert obs.tool_result["score"] > 0.5
+    assert 0.01 <= obs.tool_result["score"] <= 0.99
+    assert obs.error == ""
+    return obs
+
+
+def test_secret_exposure_success_path():
+    report = {
+        "findings": [
+            {
+                "finding_type": "secret_exposure",
+                "evidence_ids": ["EVID-101"],
+                "impact": "A synthetic API key secret is exposed in config.",
+                "remediation": "Remove the key and rotate the credential.",
+            }
+        ]
+    }
+    obs = _run_success_path(
+        "secret_exposure_easy",
+        [
+            CyberAnalystAction(tool_name="search_repo", args={"query": "api key"}),
+            CyberAnalystAction(
+                tool_name="create_finding",
+                args={
+                    "finding_type": "secret_exposure",
+                    "evidence_ids": ["EVID-101"],
+                    "severity_guess": "high",
+                    "remediation": "Remove and rotate the synthetic credential.",
+                },
+            ),
+            CyberAnalystAction(tool_name="validate_finding", args={"finding_id": "FND-001"}),
+            CyberAnalystAction(tool_name="submit_report", args={"report_json": report}),
+        ],
+    )
+    assert obs.verified_findings[0]["matching_gt_id"] == "GT-SECRET-001"
+    assert "trajectory_jsonl" in obs.tool_result
+    assert "search_repo" in obs.tool_result["trajectory_jsonl"]
+
+
+def test_missing_security_headers_success_path():
+    report = {
+        "findings": [
+            {
+                "finding_type": "missing_security_headers",
+                "evidence_ids": ["EVID-201"],
+                "impact": "The gateway is missing HSTS and CSP headers.",
+                "remediation": "Add HSTS and CSP at the gateway.",
+            }
+        ]
+    }
+    obs = _run_success_path(
+        "missing_security_headers_medium",
+        [
+            CyberAnalystAction(
+                tool_name="check_security_headers", args={"service_id": "gateway"}
+            ),
+            CyberAnalystAction(
+                tool_name="create_finding",
+                args={
+                    "finding_type": "missing_security_headers",
+                    "evidence_ids": ["EVID-201"],
+                    "severity_guess": "medium",
+                    "remediation": "Add HSTS and CSP headers.",
+                },
+            ),
+            CyberAnalystAction(tool_name="validate_finding", args={"finding_id": "FND-001"}),
+            CyberAnalystAction(tool_name="submit_report", args={"report_json": report}),
+        ],
+    )
+    assert obs.score_breakdown["valid_evidence"] == 0.15
+
+
+def test_authz_boundary_success_path_with_alias_compatible_service_ids():
+    report = {
+        "findings": [
+            {
+                "finding_type": "authz_boundary_misconfiguration",
+                "evidence_ids": ["EVID-301", "EVID-302"],
+                "impact": "The admin route authorization policy allows an analyst role.",
+                "remediation": "Apply least privilege in the policy and add a regression test.",
+            }
+        ]
+    }
+    obs = _run_success_path(
+        "authz_boundary_hard",
+        [
+            CyberAnalystAction(tool_name="list_assets", args={}),
+            CyberAnalystAction(
+                tool_name="get_log_events",
+                args={"service_id": "admin-service", "query": "admin export"},
+            ),
+            CyberAnalystAction(tool_name="search_repo", args={"query": "admin export"}),
+            CyberAnalystAction(
+                tool_name="create_finding",
+                args={
+                    "finding_type": "authz_boundary_misconfiguration",
+                    "evidence_ids": ["EVID-301", "EVID-302"],
+                    "severity_guess": "critical",
+                    "remediation": "Apply least privilege and add a regression test.",
+                },
+            ),
+            CyberAnalystAction(tool_name="validate_finding", args={"finding_id": "FND-001"}),
+            CyberAnalystAction(tool_name="submit_report", args={"report_json": report}),
+        ],
+    )
+    assert obs.score_breakdown["actionable_remediation"] == 0.15
+
+
+def test_invalid_tool_returns_observation_error():
+    env = CyberAnalystEnvironment()
+    env.reset(task_id="secret_exposure_easy", seed=1)
+    obs = env.step(CyberAnalystAction(tool_name="shell", args={"cmd": "whoami"}))
+    assert obs.done is False
+    assert obs.error == "unsupported_tool"
+    assert obs.tool_result["ok"] is False
+
+
+def test_hallucinated_report_scores_low_but_in_range():
+    env = CyberAnalystEnvironment()
+    env.reset(task_id="secret_exposure_easy", seed=1)
+    obs = env.step(
+        CyberAnalystAction(
+            tool_name="submit_report",
+            args={
+                "report_json": {
+                    "findings": [
+                        {
+                            "finding_type": "remote_code_execution",
+                            "evidence_ids": [],
+                            "impact": "Unsupported claim.",
+                            "remediation": "Unsupported remediation.",
+                        }
+                    ]
+                }
+            },
+        )
+    )
+    assert obs.done is True
+    assert obs.tool_result["score"] == 0.01
+
+
+def test_repeated_action_hard_stops_episode():
+    env = CyberAnalystEnvironment()
+    env.reset(task_id="secret_exposure_easy", seed=1)
+    obs = None
+    for _ in range(6):
+        obs = env.step(CyberAnalystAction(tool_name="list_assets", args={}))
+    assert obs is not None
+    assert obs.done is True
+    assert obs.error == "repeat_hard_stop"
+
+
+def test_seed_determinism_for_assets():
+    env_one = CyberAnalystEnvironment()
+    env_two = CyberAnalystEnvironment()
+    env_one.reset(task_id="authz_boundary_hard", seed=22)
+    env_two.reset(task_id="authz_boundary_hard", seed=22)
+    obs_one = env_one.step(CyberAnalystAction(tool_name="list_assets", args={}))
+    obs_two = env_two.step(CyberAnalystAction(tool_name="list_assets", args={}))
+    assert obs_one.tool_result == obs_two.tool_result
+
+
+def test_grader_adapters_and_clamp_are_strictly_in_range():
+    assert safe_reward(-1) == 0.01
+    assert safe_reward(2) == 0.99
+    assert 0.01 <= grade_secret_exposure_easy() <= 0.99
+    assert 0.01 <= grade_missing_security_headers_medium() <= 0.99
+    assert 0.01 <= grade_authz_boundary_hard() <= 0.99

tests/test_inference.py

@@ -1,47 +1,47 @@
-import pytest
-
-from Cyber_analyst.inference import (
-    ModelActionError,
-    action_to_log,
-    error_action,
-    parse_model_action,
-)
-
-
-def test_parse_model_action_accepts_compact_json():
-    action = parse_model_action('{"tool_name":"search_repo","args":{"query":"api key"}}')
-
-    assert action.tool_name == "search_repo"
-    assert action.args == {"query": "api key"}
-
-
-def test_parse_model_action_accepts_fenced_json():
-    action = parse_model_action(
-        """```json
-{"tool_name":"list_assets","args":{}}
-```"""
-    )
-
-    assert action.tool_name == "list_assets"
-    assert action.args == {}
-
-
-def test_parse_model_action_rejects_malformed_json():
-    with pytest.raises(ModelActionError, match="model_parse_error"):
-        parse_model_action("search the repo for api keys")
-
-
-def test_action_to_log_is_single_line_json():
-    action = parse_model_action('{"tool_name":"search_repo","args":{"query":"api\\nkey"}}')
-
-    logged = action_to_log(action)
-
-    assert "\n" not in logged
-    assert logged == '{"args":{"query":"api\\nkey"},"tool_name":"search_repo"}'
-
-
-def test_error_action_uses_strict_diagnostic_tool_name():
-    action = error_action(ModelActionError("model_parse_error: empty response"))
-
-    assert action.tool_name == "model_parse_error"
-    assert action.args == {"message": "model_parse_error: empty response"}
+import pytest
+
+from Cyber_analyst.inference import (
+    ModelActionError,
+    action_to_log,
+    error_action,
+    parse_model_action,
+)
+
+
+def test_parse_model_action_accepts_compact_json():
+    action = parse_model_action('{"tool_name":"search_repo","args":{"query":"api key"}}')
+
+    assert action.tool_name == "search_repo"
+    assert action.args == {"query": "api key"}
+
+
+def test_parse_model_action_accepts_fenced_json():
+    action = parse_model_action(
+        """```json
+{"tool_name":"list_assets","args":{}}
+```"""
+    )
+
+    assert action.tool_name == "list_assets"
+    assert action.args == {}
+
+
+def test_parse_model_action_rejects_malformed_json():
+    with pytest.raises(ModelActionError, match="model_parse_error"):
+        parse_model_action("search the repo for api keys")
+
+
+def test_action_to_log_is_single_line_json():
+    action = parse_model_action('{"tool_name":"search_repo","args":{"query":"api\\nkey"}}')
+
+    logged = action_to_log(action)
+
+    assert "\n" not in logged
+    assert logged == '{"args":{"query":"api\\nkey"},"tool_name":"search_repo"}'
+
+
+def test_error_action_uses_strict_diagnostic_tool_name():
+    action = error_action(ModelActionError("model_parse_error: empty response"))
+
+    assert action.tool_name == "model_parse_error"
+    assert action.args == {"message": "model_parse_error: empty response"}