feat: implement core RL training infrastructure, including GRPO training, evaluation utilities, custom environments, and Modal-based execution scripts.

README.md

@@ -1,6 +1,6 @@
 ---
-title: Cybersecurity Owasp Environment Server
-emoji: 🎾
+title: CyberSecurity_OWASP Environment Server
+emoji: 🛡️
 colorFrom: blue
 colorTo: gray
 sdk: docker
@@ -9,247 +9,156 @@ app_port: 8000
 base_path: /web
 tags:
   - openenv
+  - cybersecurity
+  - owasp
 ---
 
-# Cybersecurity Owasp Environment
+# CyberSecurity_OWASP
 
-A simple test environment that echoes back messages. Perfect for testing the env APIs as well as demonstrating environment usage patterns.
+`CyberSecurity_OWASP` is an OpenEnv-compliant reinforcement-learning environment for a single LLM agent that performs a defensive authorization-repair workflow:
 
-## Quick Start
-
-The simplest way to use the Cybersecurity Owasp environment is through the `CybersecurityOwaspEnv` class:
-
-```python
-from CyberSecurity_OWASP import CybersecurityOwaspAction, CybersecurityOwaspEnv
-
-try:
-    # Create environment from Docker image
-    CyberSecurity_OWASPenv = CybersecurityOwaspEnv.from_docker_image("CyberSecurity_OWASP-env:latest")
-
-    # Reset
-    result = CyberSecurity_OWASPenv.reset()
-    print(f"Reset: {result.observation.echoed_message}")
-
-    # Send multiple messages
-    messages = ["Hello, World!", "Testing echo", "Final message"]
-
-    for msg in messages:
-        result = CyberSecurity_OWASPenv.step(CybersecurityOwaspAction(message=msg))
-        print(f"Sent: '{msg}'")
-        print(f"  → Echoed: '{result.observation.echoed_message}'")
-        print(f"  → Length: {result.observation.message_length}")
-        print(f"  → Reward: {result.reward}")
-
-finally:
-    # Always clean up
-    CyberSecurity_OWASPenv.close()
+```text
+inspect generated app + policy -> discover authorization bug -> submit finding -> patch code -> preserve intended behavior
 ```
 
-That's it! The `CybersecurityOwaspEnv.from_docker_image()` method handles:
-- Starting the Docker container
-- Waiting for the server to be ready
-- Connecting to the environment
-- Container cleanup when you call `close()`
-
-## Building the Docker Image
+The current implementation includes a functional MVP scenario: an invoices FastAPI-style app with one injected OWASP A01 BOLA/IDOR defect, visible tests, hidden deterministic verifier checks, anti-cheat safeguards, and decomposed reward.
 
-Before using the environment, you need to build the Docker image:
+## Quick Start
 
 ```bash
-# From project root
-docker build -t CyberSecurity_OWASP-env:latest -f server/Dockerfile .
+uv sync --extra dev
+uv run --extra dev pytest
+uv run server --port 8000
 ```
 
-## Deploying to Hugging Face Spaces
-
-You can easily deploy your OpenEnv environment to Hugging Face Spaces using the `openenv push` command:
+Then connect with the OpenEnv client:
 
-```bash
-# From the environment directory (where openenv.yaml is located)
-openenv push
+```python
+from CyberSecurity_OWASP import CyberSecurityOWASPAction, CyberSecurityOWASPEnv
 
-# Or specify options
-openenv push --namespace my-org --private
+with CyberSecurityOWASPEnv(base_url="http://localhost:8000") as env:
+    result = env.reset(seed=7)
+    print(result.observation.task_brief)
+    result = env.step(CyberSecurityOWASPAction(tool_name="list_routes"))
+    print(result.observation.last_tool_result)
 ```
 
-The `openenv push` command will:
-1. Validate that the directory is an OpenEnv environment (checks for `openenv.yaml`)
-2. Prepare a custom build for Hugging Face Docker space (enables web interface)
-3. Upload to Hugging Face (ensuring you're logged in)
-
-### Prerequisites
+## Action Space
 
-- Authenticate with Hugging Face: The command will prompt for login if not already authenticated
+The agent emits one JSON action at a time:
 
-### Options
+```json
+{"tool_name":"read_file","arguments":{"path":"app/routes/invoices.py"}}
+```
 
-- `--directory`, `-d`: Directory containing the OpenEnv environment (defaults to current directory)
-- `--repo-id`, `-r`: Repository ID in format 'username/repo-name' (defaults to 'username/env-name' from openenv.yaml)
-- `--base-image`, `-b`: Base Docker image to use (overrides Dockerfile FROM)
-- `--private`: Deploy the space as private (default: public)
+Supported tools:
 
-### Examples
+- `inspect_policy_graph`
+- `list_routes`
+- `read_openapi`
+- `read_file`
+- `search_code`
+- `send_local_request`
+- `compare_identities`
+- `submit_finding`
+- `patch_file`
+- `run_visible_tests`
+- `submit_fix`
+- `noop`
 
-```bash
-# Push to your personal namespace (defaults to username/env-name from openenv.yaml)
-openenv push
+Tools are phase-gated:
 
-# Push to a specific repository
-openenv push --repo-id my-org/my-env
+- `discover`: inspect policy/routes/files, run safe local requests, compare identities, submit finding.
+- `patch`: read/search, patch editable app files, run visible tests, submit final fix.
+- `done`: stable terminal observation only.
 
-# Push with a custom base image
-openenv push --base-image ghcr.io/meta-pytorch/openenv-base:latest
+## Reward
 
-# Push as a private space
-openenv push --private
+Terminal reward uses stable components:
 
-# Combine options
-openenv push --repo-id my-org/my-env --base-image custom-base:latest --private
+```python
+{
+    "discovery": 0.0,
+    "security": 0.0,
+    "regression": 0.0,
+    "public_routes": 0.0,
+    "patch_quality": 0.0,
+    "visible_tests": 0.0,
+    "safety": 0.0,
+    "anti_cheat": 0.0,
+    "total": 0.0,
+}
 ```
 
-After deployment, your space will be available at:
-`https://huggingface.co/spaces/<repo-id>`
-
-The deployed space includes:
-- **Web Interface** at `/web` - Interactive UI for exploring the environment
-- **API Documentation** at `/docs` - Full OpenAPI/Swagger interface
-- **Health Check** at `/health` - Container health monitoring
-- **WebSocket** at `/ws` - Persistent session endpoint for low-latency interactions
+The verifier rewards blocking the hidden exploit while preserving legitimate owner/admin behavior and intentionally public routes. It penalizes deny-all fixes, hardcoded IDs, hidden file probes, external URL attempts, and test/fixture tampering.
 
-## Environment Details
+## Scenario Generation
 
-### Action
-**CybersecurityOwaspAction**: Contains a single field
-- `message` (str) - The message to echo back
+`reset(seed)` compiles a fresh isolated workspace under a temp directory. The MVP compiler generates:
 
-### Observation
-**CybersecurityOwaspObservation**: Contains the echo response and metadata
-- `echoed_message` (str) - The message echoed back
-- `message_length` (int) - Length of the message
-- `reward` (float) - Reward based on message length (length × 0.1)
-- `done` (bool) - Always False for echo environment
-- `metadata` (dict) - Additional info like step count
+- invoices domain policy graph;
+- randomized users, tenants, invoices, and IDs;
+- generated app files under `app/`;
+- visible tests under `tests/test_visible.py`;
+- hidden facts kept only in state for deterministic verification.
 
-### Reward
-The reward is calculated as: `message_length × 0.1`
-- "Hi" → reward: 0.2
-- "Hello, World!" → reward: 1.3
-- Empty message → reward: 0.0
+Additional domains and bug families are scaffolded for extension.
 
-## Advanced Usage
+## Testing
 
-### Connecting to an Existing Server
-
-If you already have a Cybersecurity Owasp environment server running, you can connect directly:
-
-```python
-from CyberSecurity_OWASP import CybersecurityOwaspEnv
-
-# Connect to existing server
-CyberSecurity_OWASPenv = CybersecurityOwaspEnv(base_url="<ENV_HTTP_URL_HERE>")
-
-# Use as normal
-result = CyberSecurity_OWASPenv.reset()
-result = CyberSecurity_OWASPenv.step(CybersecurityOwaspAction(message="Hello!"))
+```bash
+uv run --extra dev pytest
 ```
 
-Note: When connecting to an existing server, `CyberSecurity_OWASPenv.close()` will NOT stop the server.
+The suite covers model serialization, reset/step/state behavior, seed reproducibility, invalid actions, reward outcomes, anti-cheat checks, and scripted rollout policies.
 
-### Using the Context Manager
+## Training Scaffold
 
-The client supports context manager usage for automatic connection management:
+Training files are under `training/`:
 
-```python
-from CyberSecurity_OWASP import CybersecurityOwaspAction, CybersecurityOwaspEnv
-
-# Connect with context manager (auto-connects and closes)
-with CybersecurityOwaspEnv(base_url="http://localhost:8000") as env:
-    result = env.reset()
-    print(f"Reset: {result.observation.echoed_message}")
-    # Multiple steps with low latency
-    for msg in ["Hello", "World", "!"]:
-        result = env.step(CybersecurityOwaspAction(message=msg))
-        print(f"Echoed: {result.observation.echoed_message}")
-```
+- `rollout.py`
+- `reward_funcs.py`
+- `train_grpo.py`
+- `eval_before_after.py`
+- `trackio_utils.py`
+- `configs/grpo_small.yaml`
 
-The client uses WebSocket connections for:
-- **Lower latency**: No HTTP connection overhead per request
-- **Persistent session**: Server maintains your environment state
-- **Efficient for episodes**: Better for many sequential steps
+The training scaffold is intentionally minimal until the environment/verifier behavior is stable. Trackio metric names and GRPO defaults follow the project brief.
 
-### Concurrent WebSocket Sessions
+## Modal Ephemeral Runs
 
-The server supports multiple concurrent WebSocket connections. To enable this,
-modify `server/app.py` to use factory mode:
+Modal Labs support is kept in a separate launcher script so the local OpenEnv server and core training scaffold stay unchanged.
 
-```python
-# In server/app.py - use factory mode for concurrent sessions
-app = create_app(
-    CybersecurityOwaspEnvironment,  # Pass class, not instance
-    CybersecurityOwaspAction,
-    CybersecurityOwaspObservation,
-    max_concurrent_envs=4,  # Allow 4 concurrent sessions
-)
+Install the optional local Modal client:
+
+```bash
+uv sync --extra modal
 ```
 
-Then multiple clients can connect simultaneously:
+Run a temporary Modal app for a cheap environment/training smoke check:
 
-```python
-from CyberSecurity_OWASP import CybersecurityOwaspAction, CybersecurityOwaspEnv
-from concurrent.futures import ThreadPoolExecutor
-
-def run_episode(client_id: int):
-    with CybersecurityOwaspEnv(base_url="http://localhost:8000") as env:
-        result = env.reset()
-        for i in range(10):
-            result = env.step(CybersecurityOwaspAction(message=f"Client {client_id}, step {i}"))
-        return client_id, result.observation.message_length
-
-# Run 4 episodes concurrently
-with ThreadPoolExecutor(max_workers=4) as executor:
-    results = list(executor.map(run_episode, range(4)))
+```bash
+uv run --extra modal modal run scripts/modal_ephemeral_train.py --mode smoke --episodes 4
 ```
 
-## Development & Testing
-
-### Direct Environment Testing
+The app is ephemeral: Modal starts it for the command and stops it when the command exits. The remote result is written locally under `outputs/rollouts/`.
 
-Test the environment logic directly without starting the HTTP server:
+You can also validate the GRPO config construction remotely:
 
 ```bash
-# From the server directory
-python3 server/CyberSecurity_OWASP_environment.py
+uv run --extra modal modal run scripts/modal_ephemeral_train.py --mode grpo-config
 ```
 
-This verifies that:
-- Environment resets correctly
-- Step executes actions properly
-- State tracking works
-- Rewards are calculated correctly
-
-### Running Locally
-
-Run the server locally for development:
+The shell wrapper is equivalent:
 
 ```bash
-uvicorn server.app:app --reload
+MODE=smoke EPISODES=4 uv run --extra modal bash scripts/modal_run_ephemeral.sh
 ```
 
-## Project Structure
+## Docker / Spaces
 
-```
-CyberSecurity_OWASP/
-├── .dockerignore         # Docker build exclusions
-├── __init__.py            # Module exports
-├── README.md              # This file
-├── openenv.yaml           # OpenEnv manifest
-├── pyproject.toml         # Project metadata and dependencies
-├── uv.lock                # Locked dependencies (generated)
-├── client.py              # CybersecurityOwaspEnv client
-├── models.py              # Action and Observation models
-└── server/
-    ├── __init__.py        # Server module exports
-    ├── CyberSecurity_OWASP_environment.py  # Core environment logic
-    ├── app.py             # FastAPI application (HTTP + WebSocket endpoints)
-    └── Dockerfile         # Container image definition
+```bash
+docker build -t CyberSecurity_OWASP:latest -f server/Dockerfile .
+docker run --rm -p 8000:8000 CyberSecurity_OWASP:latest
+openenv push --repo-id <username>/CyberSecurity_OWASP
 ```

__init__.py

@@ -1,16 +1,22 @@
-# Copyright (c) Meta Platforms, Inc. and affiliates.
-# All rights reserved.
-#
-# This source code is licensed under the BSD-style license found in the
-# LICENSE file in the root directory of this source tree.
+"""CyberSecurity_OWASP OpenEnv package."""
 
-"""Cybersecurity Owasp Environment."""
-
-from .client import CybersecurityOwaspEnv
-from .models import CybersecurityOwaspAction, CybersecurityOwaspObservation
+from .client import CyberSecurityOWASPEnv, CybersecurityOwaspEnv
+from .models import (
+    CyberSecurityOWASPAction,
+    CyberSecurityOWASPObservation,
+    CyberSecurityOWASPState,
+    CybersecurityOwaspAction,
+    CybersecurityOwaspObservation,
+    CybersecurityOwaspState,
+)
 
 __all__ = [
+    "CyberSecurityOWASPAction",
+    "CyberSecurityOWASPObservation",
+    "CyberSecurityOWASPState",
+    "CyberSecurityOWASPEnv",
     "CybersecurityOwaspAction",
     "CybersecurityOwaspObservation",
+    "CybersecurityOwaspState",
     "CybersecurityOwaspEnv",
 ]

bug_mutator.py

@@ -0,0 +1,17 @@
+"""Bug-family metadata for generated authorization defects."""
+
+BUG_FAMILIES = {
+    "bola_idor": {
+        "name": "BOLA/IDOR",
+        "defect": "Invoice lookup returns any invoice to any authenticated user.",
+        "repair": "Require same tenant and either owner or billing_admin.",
+    },
+    "bfla": {"name": "BFLA", "status": "scaffolded"},
+    "tenant_leak": {"name": "Tenant leak", "status": "scaffolded"},
+    "jwt_claim_trust": {"name": "JWT claim trust", "status": "scaffolded"},
+    "public_route_trap": {"name": "Public route trap", "status": "scaffolded"},
+}
+
+
+def describe_bug_family(name: str) -> dict:
+    return BUG_FAMILIES.get(name, {"name": name, "status": "unknown"})

client.py

@@ -1,99 +1,39 @@
-# Copyright (c) Meta Platforms, Inc. and affiliates.
-# All rights reserved.
-#
-# This source code is licensed under the BSD-style license found in the
-# LICENSE file in the root directory of this source tree.
+"""CyberSecurity_OWASP OpenEnv client."""
 
-"""Cybersecurity Owasp Environment Client."""
+from __future__ import annotations
 
-from typing import Dict
+from typing import Any
 
 from openenv.core import EnvClient
 from openenv.core.client_types import StepResult
-from openenv.core.env_server.types import State
 
-from .models import CybersecurityOwaspAction, CybersecurityOwaspObservation
+from .models import (
+    CyberSecurityOWASPAction,
+    CyberSecurityOWASPObservation,
+    CyberSecurityOWASPState,
+)
 
 
-class CybersecurityOwaspEnv(
-    EnvClient[CybersecurityOwaspAction, CybersecurityOwaspObservation, State]
+class CyberSecurityOWASPEnv(
+    EnvClient[CyberSecurityOWASPAction, CyberSecurityOWASPObservation, CyberSecurityOWASPState]
 ):
-    """
-    Client for the Cybersecurity Owasp Environment.
+    """WebSocket client for the CyberSecurity_OWASP environment."""
 
-    This client maintains a persistent WebSocket connection to the environment server,
-    enabling efficient multi-step interactions with lower latency.
-    Each client instance has its own dedicated environment session on the server.
+    def _step_payload(self, action: CyberSecurityOWASPAction) -> dict[str, Any]:
+        return action.model_dump()
 
-    Example:
-        >>> # Connect to a running server
-        >>> with CybersecurityOwaspEnv(base_url="http://localhost:8000") as client:
-        ...     result = client.reset()
-        ...     print(result.observation.echoed_message)
-        ...
-        ...     result = client.step(CybersecurityOwaspAction(message="Hello!"))
-        ...     print(result.observation.echoed_message)
-
-    Example with Docker:
-        >>> # Automatically start container and connect
-        >>> client = CybersecurityOwaspEnv.from_docker_image("CyberSecurity_OWASP-env:latest")
-        >>> try:
-        ...     result = client.reset()
-        ...     result = client.step(CybersecurityOwaspAction(message="Test"))
-        ... finally:
-        ...     client.close()
-    """
-
-    def _step_payload(self, action: CybersecurityOwaspAction) -> Dict:
-        """
-        Convert CybersecurityOwaspAction to JSON payload for step message.
-
-        Args:
-            action: CybersecurityOwaspAction instance
-
-        Returns:
-            Dictionary representation suitable for JSON encoding
-        """
-        return {
-            "message": action.message,
-        }
-
-    def _parse_result(self, payload: Dict) -> StepResult[CybersecurityOwaspObservation]:
-        """
-        Parse server response into StepResult[CybersecurityOwaspObservation].
-
-        Args:
-            payload: JSON response data from server
-
-        Returns:
-            StepResult with CybersecurityOwaspObservation
-        """
+    def _parse_result(self, payload: dict[str, Any]) -> StepResult[CyberSecurityOWASPObservation]:
         obs_data = payload.get("observation", {})
-        observation = CybersecurityOwaspObservation(
-            echoed_message=obs_data.get("echoed_message", ""),
-            message_length=obs_data.get("message_length", 0),
-            done=payload.get("done", False),
-            reward=payload.get("reward"),
-            metadata=obs_data.get("metadata", {}),
-        )
-
+        observation = CyberSecurityOWASPObservation(**obs_data)
         return StepResult(
             observation=observation,
-            reward=payload.get("reward"),
-            done=payload.get("done", False),
+            reward=payload.get("reward", observation.reward),
+            done=payload.get("done", observation.done),
         )
 
-    def _parse_state(self, payload: Dict) -> State:
-        """
-        Parse server response into State object.
+    def _parse_state(self, payload: dict[str, Any]) -> CyberSecurityOWASPState:
+        return CyberSecurityOWASPState(**payload)
 
-        Args:
-            payload: JSON response from state request
 
-        Returns:
-            State object with episode_id and step_count
-        """
-        return State(
-            episode_id=payload.get("episode_id"),
-            step_count=payload.get("step_count", 0),
-        )
+# Backward-compatible alias from generated scaffold.
+CybersecurityOwaspEnv = CyberSecurityOWASPEnv

evals.py

@@ -0,0 +1,63 @@
+"""Small scripted policies and eval helpers for tests and demos."""
+
+from __future__ import annotations
+
+import difflib
+from typing import Iterable
+
+from .models import CyberSecurityOWASPAction
+
+
+def random_policy() -> Iterable[CyberSecurityOWASPAction]:
+    yield CyberSecurityOWASPAction(tool_name="noop")
+    yield CyberSecurityOWASPAction(tool_name="list_routes")
+    yield CyberSecurityOWASPAction(tool_name="submit_fix")
+
+
+def bad_policy() -> Iterable[CyberSecurityOWASPAction]:
+    yield CyberSecurityOWASPAction(tool_name="read_file", arguments={"path": "../hidden_tests.py"})
+    yield CyberSecurityOWASPAction(
+        tool_name="send_local_request", arguments={"method": "GET", "path": "https://example.com/"}
+    )
+    yield CyberSecurityOWASPAction(tool_name="submit_fix")
+
+
+def oracle_policy(original_source: str) -> list[CyberSecurityOWASPAction]:
+    fixed = original_source.replace(
+        "    # BUG: this only checks that the caller is authenticated. It forgets the\n"
+        "    # owner/admin and tenant policy checks required by the policy graph.\n"
+        "    return {\"status\": 200, \"body\": invoice}\n",
+        "    if invoice[\"tenant_id\"] != actor[\"tenant_id\"]:\n"
+        "        return {\"status\": 403, \"body\": {\"detail\": \"forbidden\"}}\n"
+        "    if invoice[\"owner_user_id\"] != actor[\"user_id\"] and not is_billing_admin(actor):\n"
+        "        return {\"status\": 403, \"body\": {\"detail\": \"forbidden\"}}\n"
+        "    return {\"status\": 200, \"body\": invoice}\n",
+    )
+    diff = "".join(
+        difflib.unified_diff(
+            original_source.splitlines(True),
+            fixed.splitlines(True),
+            fromfile="app/routes/invoices.py",
+            tofile="app/routes/invoices.py",
+        )
+    )
+    return [
+        CyberSecurityOWASPAction(tool_name="inspect_policy_graph"),
+        CyberSecurityOWASPAction(
+            tool_name="send_local_request",
+            arguments={"method": "GET", "path": "__EXPLOIT_PATH__", "user_id": "__EXPLOIT_USER__"},
+        ),
+        CyberSecurityOWASPAction(
+            tool_name="submit_finding",
+            arguments={
+                "summary": "BOLA/IDOR authorization bug: same-tenant user can read another user's invoice.",
+                "evidence": "__EVIDENCE__",
+                "policy_rule": "Only the owner or billing_admin in the same tenant may read invoices.",
+            },
+        ),
+        CyberSecurityOWASPAction(
+            tool_name="patch_file", arguments={"path": "app/routes/invoices.py", "diff": diff}
+        ),
+        CyberSecurityOWASPAction(tool_name="run_visible_tests"),
+        CyberSecurityOWASPAction(tool_name="submit_fix"),
+    ]

fixture_generator.py

@@ -0,0 +1,17 @@
+"""Fixture helpers for scenario compilers."""
+
+from __future__ import annotations
+
+from typing import Any
+
+
+def visible_workspace_summary(files: list[str], public_hint: dict[str, Any]) -> dict[str, Any]:
+    return {
+        "framework": "fastapi_style_python",
+        "editable_files": files,
+        "routes": [
+            {"method": "GET", "path": "/health", "public": True},
+            {"method": "GET", "path": "/invoices/{invoice_id}", "public": False},
+        ],
+        "domain": public_hint.get("domain", "invoices"),
+    }

models.py

@@ -1,27 +1,81 @@
-# Copyright (c) Meta Platforms, Inc. and affiliates.
-# All rights reserved.
-#
-# This source code is licensed under the BSD-style license found in the
-# LICENSE file in the root directory of this source tree.
+"""Typed OpenEnv models for the CyberSecurity_OWASP environment."""
 
-"""
-Data models for the Cybersecurity Owasp Environment.
+from typing import Any, Literal
 
-The CyberSecurity_OWASP environment is a simple test environment that echoes back messages.
-"""
-
-from openenv.core.env_server.types import Action, Observation
+from openenv.core.env_server.types import Action, Observation, State
 from pydantic import Field
 
 
-class CybersecurityOwaspAction(Action):
-    """Action for the Cybersecurity Owasp environment - just a message to echo."""
+CyberSecurityOWASPPhase = Literal["discover", "patch", "done"]
+CyberSecurityOWASPSplit = Literal["train", "validation", "hidden_eval"]
+
+
+class CyberSecurityOWASPAction(Action):
+    """One typed action emitted by the single defensive AppSec agent."""
+
+    tool_name: Literal[
+        "inspect_policy_graph",
+        "list_routes",
+        "read_openapi",
+        "read_file",
+        "search_code",
+        "send_local_request",
+        "compare_identities",
+        "submit_finding",
+        "patch_file",
+        "run_visible_tests",
+        "submit_fix",
+        "noop",
+    ] = Field(..., description="Tool to execute for this step")
+    arguments: dict[str, Any] = Field(
+        default_factory=dict, description="JSON-serializable tool arguments"
+    )
+
+
+class CyberSecurityOWASPObservation(Observation):
+    """Structured observation returned after reset and every action."""
+
+    phase: CyberSecurityOWASPPhase = "discover"
+    message: str = ""
+    task_brief: str = ""
+    visible_policy_hint: dict[str, Any] = Field(default_factory=dict)
+    workspace_summary: dict[str, Any] = Field(default_factory=dict)
+    available_actions: list[str] = Field(default_factory=list)
+    last_tool_result: str = ""
+    last_action_valid: bool = True
+    last_action_error: str | None = None
+    visible_test_result: str | None = None
+    reward_breakdown: dict[str, float] = Field(default_factory=dict)
+    done_reason: str | None = None
+
 
-    message: str = Field(..., description="Message to echo back")
+class CyberSecurityOWASPState(State):
+    """Internal state used for replay, validation, reward, and eval logging."""
 
+    task_id: str = ""
+    seed: int = 0
+    split: CyberSecurityOWASPSplit = "train"
+    difficulty: int = 0
+    domain: str = ""
+    bug_family: str = ""
+    phase: CyberSecurityOWASPPhase = "discover"
+    max_steps: int = 40
+    done: bool = False
+    success: bool = False
+    failure_reason: str | None = None
+    finding_submitted: bool = False
+    patch_submitted: bool = False
+    accumulated_reward: float = 0.0
+    last_reward: float = 0.0
+    action_history: list[dict[str, Any]] = Field(default_factory=list)
+    reward_history: list[dict[str, float]] = Field(default_factory=list)
+    visible_facts: dict[str, Any] = Field(default_factory=dict)
+    hidden_facts: dict[str, Any] = Field(default_factory=dict)
+    metrics: dict[str, Any] = Field(default_factory=dict)
+    anti_cheat_flags: list[str] = Field(default_factory=list)
 
-class CybersecurityOwaspObservation(Observation):
-    """Observation from the Cybersecurity Owasp environment - the echoed message."""
 
-    echoed_message: str = Field(default="", description="The echoed message")
-    message_length: int = Field(default=0, description="Length of the echoed message")
+# Backward-compatible aliases from the OpenEnv scaffold.
+CybersecurityOwaspAction = CyberSecurityOWASPAction
+CybersecurityOwaspObservation = CyberSecurityOWASPObservation
+CybersecurityOwaspState = CyberSecurityOWASPState

policy_graph.py

@@ -0,0 +1,105 @@
+"""Policy graph generation for MVP authorization-repair scenarios."""
+
+from __future__ import annotations
+
+import random
+from dataclasses import dataclass
+from typing import Any
+
+
+@dataclass(frozen=True)
+class CompiledPolicy:
+    public_hint: dict[str, Any]
+    hidden_facts: dict[str, Any]
+
+
+def build_invoice_policy(seed: int) -> CompiledPolicy:
+    """Create a deterministic invoices policy graph with randomized IDs."""
+
+    rng = random.Random(seed)
+    suffix = rng.randint(1000, 9999)
+    tenant_a = f"tenant-{suffix}-a"
+    tenant_b = f"tenant-{suffix}-b"
+    owner = f"user-{suffix}-owner"
+    intruder = f"user-{suffix}-intruder"
+    admin = f"user-{suffix}-billing-admin"
+    owner_invoice = f"inv-{suffix}-owned"
+    other_invoice = f"inv-{suffix}-other"
+    foreign_invoice = f"inv-{suffix}-foreign"
+
+    users = {
+        owner: {"user_id": owner, "tenant_id": tenant_a, "roles": ["user"]},
+        intruder: {"user_id": intruder, "tenant_id": tenant_a, "roles": ["user"]},
+        admin: {"user_id": admin, "tenant_id": tenant_a, "roles": ["billing_admin"]},
+        f"user-{suffix}-outside": {
+            "user_id": f"user-{suffix}-outside",
+            "tenant_id": tenant_b,
+            "roles": ["user"],
+        },
+    }
+    invoices = {
+        owner_invoice: {
+            "invoice_id": owner_invoice,
+            "tenant_id": tenant_a,
+            "owner_user_id": owner,
+            "amount": 1200,
+            "status": "open",
+        },
+        other_invoice: {
+            "invoice_id": other_invoice,
+            "tenant_id": tenant_a,
+            "owner_user_id": intruder,
+            "amount": 410,
+            "status": "open",
+        },
+        foreign_invoice: {
+            "invoice_id": foreign_invoice,
+            "tenant_id": tenant_b,
+            "owner_user_id": f"user-{suffix}-outside",
+            "amount": 990,
+            "status": "open",
+        },
+    }
+
+    public_hint = {
+        "domain": "invoices",
+        "policy_rules": [
+            "Authenticated users may read invoices they own.",
+            "Billing admins may read invoices in their own tenant.",
+            "Users must not read another user's invoice unless they have a billing_admin role.",
+            "Cross-tenant invoice reads are forbidden.",
+            "GET /health is intentionally public.",
+        ],
+        "users": {
+            alias: {
+                "user_id": value["user_id"],
+                "tenant_id": value["tenant_id"],
+                "roles": value["roles"],
+            }
+            for alias, value in {
+                "owner": users[owner],
+                "same_tenant_other_user": users[intruder],
+                "billing_admin": users[admin],
+            }.items()
+        },
+        "resources": {
+            "owned_invoice": owner_invoice,
+            "same_tenant_other_invoice": other_invoice,
+            "foreign_tenant_invoice": foreign_invoice,
+        },
+        "public_routes": [{"method": "GET", "path": "/health"}],
+    }
+    hidden_facts = {
+        "users": users,
+        "invoices": invoices,
+        "owner_user_id": owner,
+        "intruder_user_id": intruder,
+        "admin_user_id": admin,
+        "owner_invoice_id": owner_invoice,
+        "other_invoice_id": other_invoice,
+        "foreign_invoice_id": foreign_invoice,
+        "tenant_a": tenant_a,
+        "tenant_b": tenant_b,
+        "bug_family": "bola_idor",
+    }
+    return CompiledPolicy(public_hint=public_hint, hidden_facts=hidden_facts)

pyproject.toml

@@ -33,6 +33,9 @@ dev = [
     "pytest>=8.0.0",
     "pytest-cov>=4.0.0",
 ]
+modal = [
+    "modal>=1.1.0",
+]
 
 [project.scripts]
 # Server entry point - enables running via: uv run --project . server
@@ -42,4 +45,4 @@ server = "CyberSecurity_OWASP.server.app:main"
 [tool.setuptools]
 include-package-data = true
 packages = ["CyberSecurity_OWASP", "CyberSecurity_OWASP.server"]
-package-dir = { "CyberSecurity_OWASP" = ".", "CyberSecurity_OWASP.server" = "server" }
+package-dir = { "CyberSecurity_OWASP" = ".", "CyberSecurity_OWASP.server" = "server" }

rewards.py

@@ -0,0 +1,66 @@
+"""Reward computation for CyberSecurity_OWASP."""
+
+from __future__ import annotations
+
+from .models import CyberSecurityOWASPAction, CyberSecurityOWASPState
+
+
+REWARD_KEYS = (
+    "discovery",
+    "security",
+    "regression",
+    "public_routes",
+    "patch_quality",
+    "visible_tests",
+    "safety",
+    "anti_cheat",
+    "total",
+)
+
+
+def empty_reward() -> dict[str, float]:
+    return {key: 0.0 for key in REWARD_KEYS}
+
+
+def compute_reward(
+    state: CyberSecurityOWASPState,
+    action: CyberSecurityOWASPAction,
+    verifier_result: dict,
+) -> dict[str, float]:
+    reward = empty_reward()
+    if action.tool_name == "submit_finding":
+        finding = verifier_result.get("finding", {})
+        reward["discovery"] = (2.0 if finding.get("valid") else 0.0) + (
+            1.0 if finding.get("tied_to_policy") else 0.0
+        )
+    elif action.tool_name == "run_visible_tests":
+        visible = verifier_result.get("visible", {})
+        reward["visible_tests"] = 1.0 if visible.get("passed") else 0.0
+    elif action.tool_name == "submit_fix":
+        security = verifier_result.get("security", {})
+        regression = verifier_result.get("regression", {})
+        public_routes = verifier_result.get("public_routes", {})
+        quality = verifier_result.get("patch_quality", {})
+        visible = verifier_result.get("visible", {})
+        reward["security"] = 5.0 if security.get("passed") else -6.0
+        reward["regression"] = 3.0 if regression.get("passed") else -5.0
+        reward["public_routes"] = 1.0 if public_routes.get("passed") else -3.0
+        reward["patch_quality"] = 2.0 if quality.get("passed") else 0.0
+        reward["visible_tests"] = 1.0 if visible.get("passed") else -6.0
+        if quality.get("hardcoded"):
+            reward["anti_cheat"] -= 4.0
+        if quality.get("deny_all"):
+            reward["safety"] -= 5.0
+
+    flags = verifier_result.get("anti_cheat_flags", []) or []
+    if flags:
+        if "external_network_attempt" in flags:
+            reward["safety"] -= 8.0
+        if "blocked_file_probe" in flags:
+            reward["anti_cheat"] -= 8.0
+        if "hardcoded_hidden_identifier" in flags:
+            reward["anti_cheat"] -= 4.0
+
+    total = sum(value for key, value in reward.items() if key != "total")
+    reward["total"] = min(15.0, total) if total > 0 else total
+    return reward

safety.py

@@ -0,0 +1,17 @@
+"""Safety boundaries for local-only defensive AppSec episodes."""
+
+from __future__ import annotations
+
+
+FORBIDDEN_BEHAVIOR = (
+    "external network access",
+    "host filesystem reads",
+    "hidden test access",
+    "oracle access",
+    "credential extraction",
+    "persistence or evasion",
+)
+
+
+def is_local_route(path: str) -> bool:
+    return path.startswith("/") and not path.startswith("//") and "://" not in path

scenario_compiler.py

@@ -0,0 +1,46 @@
+"""Deterministic scenario compiler for CyberSecurity_OWASP."""
+
+from __future__ import annotations
+
+import tempfile
+from pathlib import Path
+from typing import Any
+
+from .fixture_generator import visible_workspace_summary
+from .policy_graph import build_invoice_policy
+from .template_renderer import render_fastapi_basic
+
+
+def compile_scenario(seed: int, split: str = "train", difficulty: int = 0) -> dict[str, Any]:
+    """Compile one isolated MVP authorization-repair scenario."""
+
+    compiled = build_invoice_policy(seed)
+    workspace = Path(tempfile.mkdtemp(prefix=f"cybersecurity_owasp_{split}_{seed}_"))
+    editable_files = render_fastapi_basic(workspace, compiled.public_hint, compiled.hidden_facts)
+    task_id = f"{split}-invoices-bola-{seed}"
+    hidden = dict(compiled.hidden_facts)
+    hidden.update(
+        {
+            "workspace": str(workspace),
+            "editable_files": editable_files,
+            "initial_file_hashes": {
+                path: (workspace / path).read_text(encoding="utf-8")
+                for path in editable_files
+            },
+        }
+    )
+    return {
+        "task_id": task_id,
+        "workspace": workspace,
+        "domain": "invoices",
+        "bug_family": "bola_idor",
+        "difficulty": difficulty,
+        "task_brief": (
+            "Inspect the generated invoices app and policy. Find the broken "
+            "authorization behavior, submit a finding with local evidence, patch "
+            "the app, preserve intended owner/admin/public behavior, then submit."
+        ),
+        "public_hint": compiled.public_hint,
+        "workspace_summary": visible_workspace_summary(editable_files, compiled.public_hint),
+        "hidden_facts": hidden,
+    }

scripts/docker_build.sh

@@ -0,0 +1,3 @@
+#!/usr/bin/env bash
+set -euo pipefail
+docker build -t CyberSecurity_OWASP:latest -f server/Dockerfile .

scripts/docker_run.sh

@@ -0,0 +1,3 @@
+#!/usr/bin/env bash
+set -euo pipefail
+docker run --rm -p "${PORT:-8000}:8000" CyberSecurity_OWASP:latest

scripts/generate_scenarios.sh

@@ -0,0 +1,3 @@
+#!/usr/bin/env bash
+set -euo pipefail
+uv run python -c "from CyberSecurity_OWASP.scenario_compiler import compile_scenario; [compile_scenario(i) for i in range(3)]; print('generated 3 smoke scenarios')"

scripts/modal_ephemeral_train.py

@@ -0,0 +1,163 @@
+"""Ephemeral Modal Labs launcher for CyberSecurity_OWASP training smoke runs.
+
+Run from the repo root:
+
+    modal run scripts/modal_ephemeral_train.py --mode smoke --episodes 4
+
+This intentionally stays separate from ``training/train_grpo.py``. It packages
+the local repo into a temporary Modal app and returns compact JSON artifacts to
+the local process, so the run disappears when ``modal run`` exits.
+"""
+
+from __future__ import annotations
+
+import json
+from datetime import datetime
+from pathlib import Path
+from typing import Any
+
+import modal
+
+
+APP_NAME = "CyberSecurity_OWASP-ephemeral-training"
+REMOTE_PROJECT = "/root/CyberSecurity_OWASP"
+PROJECT_ROOT = Path(__file__).resolve().parents[1]
+
+app = modal.App(APP_NAME)
+
+image = (
+    modal.Image.debian_slim(python_version="3.11")
+    .apt_install("git")
+    .add_local_dir(
+        PROJECT_ROOT,
+        remote_path=REMOTE_PROJECT,
+        copy=True,
+        ignore=[
+            ".git",
+            ".venv",
+            "__pycache__",
+            ".pytest_cache",
+            "outputs",
+            "*.pyc",
+        ],
+    )
+    .run_commands(f"pip install -e {REMOTE_PROJECT}")
+    .workdir(REMOTE_PROJECT)
+)
+
+
+class NoopTrainer:
+    """Deterministic placeholder policy for cheap Modal smoke runs."""
+
+    def generate_rollout_completions(self, prompts: list[str]) -> list[dict[str, Any]]:
+        return [
+            {
+                "text": '{"tool_name":"noop","arguments":{}}',
+                "prompt_ids": [],
+                "completion_ids": [],
+                "logprobs": [],
+            }
+            for _ in prompts
+        ]
+
+
+@app.function(image=image, timeout=60 * 30)
+def run_ephemeral_smoke(episodes: int = 4, seed_start: int = 0) -> dict[str, Any]:
+    from CyberSecurity_OWASP.models import CyberSecurityOWASPAction
+    from CyberSecurity_OWASP.server.CyberSecurity_OWASP_environment import (
+        CybersecurityOwaspEnvironment,
+    )
+    from training.rollout import rollout_once
+
+    baseline = []
+    oracle = []
+
+    for offset in range(episodes):
+        seed = seed_start + offset
+
+        baseline_env = CybersecurityOwaspEnvironment()
+        baseline_env.reset(seed=seed, split="validation")
+        baseline.append(rollout_once(NoopTrainer(), baseline_env, max_steps=5))
+
+        oracle_env = CybersecurityOwaspEnvironment()
+        oracle_env.reset(seed=seed, split="validation")
+        hidden = oracle_env.state.hidden_facts
+        oracle_env.step(
+            CyberSecurityOWASPAction(
+                tool_name="submit_finding",
+                arguments={
+                    "summary": "BOLA/IDOR authorization bug in invoice read route.",
+                    "evidence": (
+                        f"user {hidden['owner_user_id']} can request invoice "
+                        f"{hidden['other_invoice_id']} despite the owner/admin policy"
+                    ),
+                    "policy_rule": "Only owner or billing_admin in same tenant may read invoices.",
+                },
+            )
+        )
+        source = (
+            Path(hidden["workspace"]) / "app/routes/invoices.py"
+        ).read_text(encoding="utf-8")
+        fixed = source.replace(
+            "    # BUG: this only checks that the caller is authenticated. It forgets the\n"
+            "    # owner/admin and tenant policy checks required by the policy graph.\n"
+            "    return {\"status\": 200, \"body\": invoice}\n",
+            "    if invoice[\"tenant_id\"] != actor[\"tenant_id\"]:\n"
+            "        return {\"status\": 403, \"body\": {\"detail\": \"forbidden\"}}\n"
+            "    if invoice[\"owner_user_id\"] != actor[\"user_id\"] and not is_billing_admin(actor):\n"
+            "        return {\"status\": 403, \"body\": {\"detail\": \"forbidden\"}}\n"
+            "    return {\"status\": 200, \"body\": invoice}\n",
+        )
+        oracle_env.step(
+            CyberSecurityOWASPAction(
+                tool_name="patch_file",
+                arguments={"path": "app/routes/invoices.py", "content": fixed},
+            )
+        )
+        oracle_env.step(CyberSecurityOWASPAction(tool_name="run_visible_tests"))
+        final = oracle_env.step(CyberSecurityOWASPAction(tool_name="submit_fix"))
+        oracle.append(
+            {
+                "seed": seed,
+                "success": oracle_env.state.success,
+                "reward_total": final.reward_breakdown.get("total", 0.0),
+                "reward_breakdown": final.reward_breakdown,
+            }
+        )
+
+    def mean(items: list[dict[str, Any]], key: str) -> float:
+        return sum(float(item.get(key, 0.0)) for item in items) / max(1, len(items))
+
+    return {
+        "run_name": f"{APP_NAME}-{datetime.utcnow().strftime('%Y%m%d-%H%M%S')}",
+        "mode": "smoke",
+        "episodes": episodes,
+        "seed_start": seed_start,
+        "baseline_mean_reward": mean(baseline, "reward_total"),
+        "oracle_mean_reward": mean(oracle, "reward_total"),
+        "oracle_success_rate": mean(oracle, "success"),
+        "baseline": baseline,
+        "oracle": oracle,
+    }
+
+
+@app.function(image=image, timeout=60 * 10)
+def run_grpo_config_check() -> str:
+    from training.train_grpo import build_grpo_config
+
+    return str(build_grpo_config())
+
+
+@app.local_entrypoint()
+def main(mode: str = "smoke", episodes: int = 4, seed_start: int = 0) -> None:
+    if mode == "smoke":
+        result = run_ephemeral_smoke.remote(episodes=episodes, seed_start=seed_start)
+        output_dir = PROJECT_ROOT / "outputs" / "rollouts"
+        output_dir.mkdir(parents=True, exist_ok=True)
+        output_path = output_dir / f"{result['run_name']}.json"
+        output_path.write_text(json.dumps(result, indent=2, sort_keys=True), encoding="utf-8")
+        print(json.dumps({"saved": str(output_path), **result}, indent=2, sort_keys=True))
+    elif mode == "grpo-config":
+        print(run_grpo_config_check.remote())
+    else:
+        raise ValueError("mode must be 'smoke' or 'grpo-config'")

scripts/modal_run_ephemeral.sh

@@ -0,0 +1,3 @@
+#!/usr/bin/env bash
+set -euo pipefail
+modal run scripts/modal_ephemeral_train.py --mode "${MODE:-smoke}" --episodes "${EPISODES:-4}" --seed-start "${SEED_START:-0}"

scripts/push_space.sh

@@ -0,0 +1,3 @@
+#!/usr/bin/env bash
+set -euo pipefail
+openenv push --repo-id "${HF_REPO_ID:?set HF_REPO_ID, e.g. username/CyberSecurity_OWASP}"

scripts/run_local.sh

@@ -0,0 +1,3 @@
+#!/usr/bin/env bash
+set -euo pipefail
+uv run server --port "${PORT:-8000}"

scripts/smoke_test.sh

@@ -0,0 +1,3 @@
+#!/usr/bin/env bash
+set -euo pipefail
+uv run pytest tests/test_models.py tests/test_reset_step_state.py

server/CyberSecurity_OWASP_environment.py

@@ -1,104 +1,366 @@
-# Copyright (c) Meta Platforms, Inc. and affiliates.
-# All rights reserved.
-#
-# This source code is licensed under the BSD-style license found in the
-# LICENSE file in the root directory of this source tree.
+"""CyberSecurity_OWASP OpenEnv environment implementation."""
 
-"""
-Cybersecurity Owasp Environment Implementation.
-
-A simple test environment that echoes back messages sent to it.
-Perfect for testing HTTP server infrastructure.
-"""
+from __future__ import annotations
 
+import json
+import shutil
+from pathlib import Path
+from typing import Any
 from uuid import uuid4
 
 from openenv.core.env_server.interfaces import Environment
-from openenv.core.env_server.types import State
 
 try:
-    from ..models import CybersecurityOwaspAction, CybersecurityOwaspObservation
-except ImportError:
-    from models import CybersecurityOwaspAction, CybersecurityOwaspObservation
-
-
-class CybersecurityOwaspEnvironment(Environment):
-    """
-    A simple echo environment that echoes back messages.
-
-    This environment is designed for testing the HTTP server infrastructure.
-    It maintains minimal state and simply echoes back whatever message it receives.
-
-    Example:
-        >>> env = CybersecurityOwaspEnvironment()
-        >>> obs = env.reset()
-        >>> print(obs.echoed_message)  # "Cybersecurity Owasp environment ready!"
-        >>>
-        >>> obs = env.step(CybersecurityOwaspAction(message="Hello"))
-        >>> print(obs.echoed_message)  # "Hello"
-        >>> print(obs.message_length)  # 5
-    """
-
-    # Enable concurrent WebSocket sessions.
-    # Set to True if your environment isolates state between instances.
-    # When True, multiple WebSocket clients can connect simultaneously, each
-    # getting their own environment instance (when using factory mode in app.py).
-    SUPPORTS_CONCURRENT_SESSIONS: bool = True
+    from ..models import (
+        CyberSecurityOWASPAction,
+        CyberSecurityOWASPObservation,
+        CyberSecurityOWASPState,
+    )
+    from ..scenario_compiler import compile_scenario
+    from ..safety import is_local_route
+    from ..validators import detect_cheating, is_path_allowed, simulate_request
+    from .reward_engine import evaluate_action
+except ImportError:  # pragma: no cover
+    from models import CyberSecurityOWASPAction, CyberSecurityOWASPObservation, CyberSecurityOWASPState
+    from scenario_compiler import compile_scenario
+    from safety import is_local_route
+    from validators import detect_cheating, is_path_allowed, simulate_request
+    from server.reward_engine import evaluate_action
+
+
+ALLOWED_TOOLS = {
+    "discover": {
+        "inspect_policy_graph",
+        "list_routes",
+        "read_openapi",
+        "read_file",
+        "search_code",
+        "send_local_request",
+        "compare_identities",
+        "submit_finding",
+        "noop",
+    },
+    "patch": {
+        "read_file",
+        "search_code",
+        "patch_file",
+        "run_visible_tests",
+        "send_local_request",
+        "submit_fix",
+        "noop",
+    },
+    "done": set(),
+}
+
+
+class CybersecurityOwaspEnvironment(
+    Environment[CyberSecurityOWASPAction, CyberSecurityOWASPObservation, CyberSecurityOWASPState]
+):
+    """Single-agent defensive authorization-repair environment."""
+
+    SUPPORTS_CONCURRENT_SESSIONS = True
 
     def __init__(self):
-        """Initialize the CyberSecurity_OWASP environment."""
-        self._state = State(episode_id=str(uuid4()), step_count=0)
-        self._reset_count = 0
-
-    def reset(self) -> CybersecurityOwaspObservation:
-        """
-        Reset the environment.
-
-        Returns:
-            CybersecurityOwaspObservation with a ready message
-        """
-        self._state = State(episode_id=str(uuid4()), step_count=0)
-        self._reset_count += 1
-
-        return CybersecurityOwaspObservation(
-            echoed_message="Cybersecurity Owasp environment ready!",
-            message_length=0,
+        super().__init__()
+        self._state = CyberSecurityOWASPState(episode_id=str(uuid4()))
+        self._task_brief = ""
+        self._visible_policy_hint: dict[str, Any] = {}
+        self._workspace_summary: dict[str, Any] = {}
+        self._last_done_observation: CyberSecurityOWASPObservation | None = None
+
+    def reset(
+        self,
+        seed: int | None = None,
+        episode_id: str | None = None,
+        split: str = "train",
+        difficulty: int = 0,
+        **_: Any,
+    ) -> CyberSecurityOWASPObservation:
+        self.close()
+        actual_seed = int(seed if seed is not None else 0)
+        scenario = compile_scenario(actual_seed, split=split, difficulty=difficulty)
+        self._state = CyberSecurityOWASPState(
+            episode_id=episode_id or str(uuid4()),
+            task_id=scenario["task_id"],
+            seed=actual_seed,
+            split=split,
+            difficulty=difficulty,
+            domain=scenario["domain"],
+            bug_family=scenario["bug_family"],
+            phase="discover",
+            step_count=0,
+            max_steps=40,
             done=False,
-            reward=0.0,
+            success=False,
+            visible_facts={"workspace_summary": scenario["workspace_summary"]},
+            hidden_facts=scenario["hidden_facts"],
+            metrics={"reset_count": 1},
         )
+        self._task_brief = scenario["task_brief"]
+        self._visible_policy_hint = scenario["public_hint"]
+        self._workspace_summary = scenario["workspace_summary"]
+        self._last_done_observation = None
+        return self._observation("Scenario ready. Start in discover phase.", reward=0.0)
 
-    def step(self, action: CybersecurityOwaspAction) -> CybersecurityOwaspObservation:  # type: ignore[override]
-        """
-        Execute a step in the environment by echoing the message.
+    def step(
+        self,
+        action: CyberSecurityOWASPAction,
+        timeout_s: float | None = None,
+        **_: Any,
+    ) -> CyberSecurityOWASPObservation:
+        if self._state.done:
+            return self._last_done_observation or self._observation(
+                "Episode is already done.", reward=0.0, done_reason=self._state.failure_reason
+            )
 
-        Args:
-            action: CybersecurityOwaspAction containing the message to echo
+        anti_cheat_flags = detect_cheating(self._state, action)
+        for flag in anti_cheat_flags:
+            if flag not in self._state.anti_cheat_flags:
+                self._state.anti_cheat_flags.append(flag)
 
-        Returns:
-            CybersecurityOwaspObservation with the echoed message and its length
-        """
         self._state.step_count += 1
+        self._state.action_history.append(
+            {"tool_name": action.tool_name, "arguments": action.arguments}
+        )
 
-        message = action.message
-        length = len(message)
+        if action.tool_name not in ALLOWED_TOOLS[self._state.phase]:
+            verifier, reward = evaluate_action(self._state, action, anti_cheat_flags)
+            return self._finish_step(
+                "Action is not allowed in the current phase.",
+                reward,
+                valid=False,
+                error=f"{action.tool_name} is not allowed during {self._state.phase}",
+                verifier=verifier,
+            )
 
-        # Simple reward: longer messages get higher rewards
-        reward = length * 0.1
+        try:
+            result, verifier, reward, visible_tests = self._execute(action, anti_cheat_flags)
+            return self._finish_step(
+                result,
+                reward,
+                valid=True,
+                verifier=verifier,
+                visible_test_result=visible_tests,
+            )
+        except Exception as exc:  # keep malformed agent actions from crashing the server
+            verifier, reward = evaluate_action(self._state, action, anti_cheat_flags)
+            return self._finish_step(
+                "Tool execution failed.",
+                reward,
+                valid=False,
+                error=str(exc),
+                verifier=verifier,
+            )
 
-        return CybersecurityOwaspObservation(
-            echoed_message=message,
-            message_length=length,
-            done=False,
+    @property
+    def state(self) -> CyberSecurityOWASPState:
+        return self._state
+
+    def close(self) -> None:
+        workspace = self._state.hidden_facts.get("workspace")
+        if workspace:
+            shutil.rmtree(workspace, ignore_errors=True)
+
+    def _execute(
+        self, action: CyberSecurityOWASPAction, anti_cheat_flags: list[str]
+    ) -> tuple[str, dict, dict[str, float], str | None]:
+        verifier: dict = {"anti_cheat_flags": anti_cheat_flags}
+        reward = {key: 0.0 for key in (
+            "discovery",
+            "security",
+            "regression",
+            "public_routes",
+            "patch_quality",
+            "visible_tests",
+            "safety",
+            "anti_cheat",
+            "total",
+        )}
+        visible_tests = None
+        args = action.arguments or {}
+
+        if action.tool_name == "noop":
+            return "No operation.", verifier, reward, None
+        if action.tool_name == "inspect_policy_graph":
+            return json.dumps(self._visible_policy_hint, indent=2, sort_keys=True), verifier, reward, None
+        if action.tool_name == "list_routes":
+            return json.dumps(self._workspace_summary["routes"], indent=2), verifier, reward, None
+        if action.tool_name == "read_openapi":
+            return json.dumps(
+                {
+                    "openapi": "3.1.0",
+                    "info": {"title": "Generated invoices app", "version": "0.1.0"},
+                    "paths": {
+                        "/health": {"get": {"x-public": True}},
+                        "/invoices/{invoice_id}": {"get": {"x-public": False}},
+                    },
+                },
+                indent=2,
+            ), verifier, reward, None
+        if action.tool_name == "read_file":
+            path = self._resolve_path(str(args.get("path", "")))
+            return path.read_text(encoding="utf-8"), verifier, reward, None
+        if action.tool_name == "search_code":
+            return self._search_code(str(args.get("query", ""))), verifier, reward, None
+        if action.tool_name == "send_local_request":
+            if not is_local_route(str(args.get("path", ""))):
+                raise ValueError("send_local_request only accepts local route paths")
+            response = simulate_request(
+                self._state,
+                str(args.get("method", "GET")),
+                str(args.get("path", "")),
+                args.get("user_id"),
+            )
+            return json.dumps(response, indent=2, sort_keys=True), verifier, reward, None
+        if action.tool_name == "compare_identities":
+            path = str(args.get("path", ""))
+            first = str(args.get("first_user_id", ""))
+            second = str(args.get("second_user_id", ""))
+            if not is_local_route(path):
+                raise ValueError("compare_identities only accepts local route paths")
+            response = {
+                "first": simulate_request(self._state, str(args.get("method", "GET")), path, first),
+                "second": simulate_request(self._state, str(args.get("method", "GET")), path, second),
+            }
+            return json.dumps(response, indent=2, sort_keys=True), verifier, reward, None
+        if action.tool_name == "submit_finding":
+            verifier, reward = evaluate_action(self._state, action, anti_cheat_flags)
+            if verifier.get("finding", {}).get("valid"):
+                self._state.finding_submitted = True
+                self._state.phase = "patch"
+                return "Finding accepted. Patch phase unlocked.", verifier, reward, None
+            return "Finding was not specific enough to unlock patching.", verifier, reward, None
+        if action.tool_name == "patch_file":
+            path = self._resolve_path(str(args.get("path", "")), write=True)
+            if "content" in args:
+                path.write_text(str(args["content"]), encoding="utf-8")
+            else:
+                self._apply_unified_diff(path, str(args.get("diff", "")))
+            return f"Patched {args.get('path')}.", verifier, reward, None
+        if action.tool_name == "run_visible_tests":
+            verifier, reward = evaluate_action(self._state, action, anti_cheat_flags)
+            visible_tests = json.dumps(verifier.get("visible", {}), indent=2, sort_keys=True)
+            return visible_tests, verifier, reward, visible_tests
+        if action.tool_name == "submit_fix":
+            verifier, reward = evaluate_action(self._state, action, anti_cheat_flags)
+            self._state.patch_submitted = True
+            security = verifier.get("security", {}).get("passed", False)
+            regression = verifier.get("regression", {}).get("passed", False)
+            public = verifier.get("public_routes", {}).get("passed", False)
+            quality = verifier.get("patch_quality", {}).get("passed", False)
+            self._state.success = bool(security and regression and public and quality)
+            self._state.done = True
+            self._state.phase = "done"
+            self._state.failure_reason = None if self._state.success else "hidden_verifier_failed"
+            return json.dumps(verifier, indent=2, sort_keys=True), verifier, reward, None
+        raise ValueError(f"Unhandled tool {action.tool_name}")
+
+    def _finish_step(
+        self,
+        message: str,
+        reward: dict[str, float],
+        *,
+        valid: bool,
+        error: str | None = None,
+        verifier: dict | None = None,
+        visible_test_result: str | None = None,
+    ) -> CyberSecurityOWASPObservation:
+        self._state.last_reward = float(reward.get("total", 0.0))
+        self._state.accumulated_reward += self._state.last_reward
+        self._state.reward_history.append(reward)
+        if self._state.step_count >= self._state.max_steps and not self._state.done:
+            self._state.done = True
+            self._state.phase = "done"
+            self._state.failure_reason = "max_steps_exceeded"
+        obs = self._observation(
+            message,
+            reward=self._state.last_reward,
+            valid=valid,
+            error=error,
+            reward_breakdown=reward,
+            visible_test_result=visible_test_result,
+            done_reason=self._state.failure_reason,
+        )
+        if self._state.done:
+            self._last_done_observation = obs
+        return obs
+
+    def _observation(
+        self,
+        message: str,
+        *,
+        reward: float,
+        valid: bool = True,
+        error: str | None = None,
+        reward_breakdown: dict[str, float] | None = None,
+        visible_test_result: str | None = None,
+        done_reason: str | None = None,
+    ) -> CyberSecurityOWASPObservation:
+        return CyberSecurityOWASPObservation(
+            phase=self._state.phase,
+            message=message,
+            task_brief=self._task_brief,
+            visible_policy_hint=self._visible_policy_hint,
+            workspace_summary=self._workspace_summary,
+            available_actions=sorted(ALLOWED_TOOLS[self._state.phase]),
+            last_tool_result=message,
+            last_action_valid=valid,
+            last_action_error=error,
+            visible_test_result=visible_test_result,
+            reward_breakdown=reward_breakdown or {},
+            done_reason=done_reason,
+            done=self._state.done,
             reward=reward,
-            metadata={"original_message": message, "step": self._state.step_count},
+            metadata={"episode_id": self._state.episode_id, "step_count": self._state.step_count},
         )
 
-    @property
-    def state(self) -> State:
-        """
-        Get the current environment state.
+    def _resolve_path(self, path: str, *, write: bool = False) -> Path:
+        allowed, normalized_or_error = is_path_allowed(self._state, path, write=write)
+        if not allowed:
+            raise ValueError(normalized_or_error)
+        return Path(str(self._state.hidden_facts["workspace"])) / normalized_or_error
 
-        Returns:
-            Current State with episode_id and step_count
-        """
-        return self._state
+    def _search_code(self, query: str) -> str:
+        if not query:
+            raise ValueError("query is required")
+        results: list[str] = []
+        workspace = Path(str(self._state.hidden_facts["workspace"]))
+        for rel in self._state.hidden_facts.get("editable_files", []):
+            path = workspace / rel
+            text = path.read_text(encoding="utf-8")
+            for idx, line in enumerate(text.splitlines(), start=1):
+                if query.lower() in line.lower():
+                    results.append(f"{rel}:{idx}: {line}")
+        return "\n".join(results) or "No matches."
+
+    def _apply_unified_diff(self, path: Path, diff: str) -> None:
+        if not diff.strip():
+            raise ValueError("diff or content is required")
+        original = path.read_text(encoding="utf-8").splitlines(True)
+        output: list[str] = []
+        old_index = 0
+        lines = diff.splitlines(True)
+        i = 0
+        while i < len(lines):
+            line = lines[i]
+            if not line.startswith("@@"):
+                i += 1
+                continue
+            old_start = int(line.split()[1].split(",")[0][1:])
+            output.extend(original[old_index : old_start - 1])
+            old_index = old_start - 1
+            i += 1
+            while i < len(lines) and not lines[i].startswith("@@"):
+                hunk_line = lines[i]
+                if hunk_line.startswith(" "):
+                    output.append(original[old_index])
+                    old_index += 1
+                elif hunk_line.startswith("-"):
+                    old_index += 1
+                elif hunk_line.startswith("+"):
+                    output.append(hunk_line[1:])
+                elif hunk_line.startswith("\\"):
+                    pass
+                i += 1
+        output.extend(original[old_index:])
+        path.write_text("".join(output), encoding="utf-8")

server/app.py

@@ -4,29 +4,7 @@
 # This source code is licensed under the BSD-style license found in the
 # LICENSE file in the root directory of this source tree.
 
-"""
-FastAPI application for the Cybersecurity Owasp Environment.
-
-This module creates an HTTP server that exposes the CybersecurityOwaspEnvironment
-over HTTP and WebSocket endpoints, compatible with EnvClient.
-
-Endpoints:
-    - POST /reset: Reset the environment
-    - POST /step: Execute an action
-    - GET /state: Get current environment state
-    - GET /schema: Get action/observation schemas
-    - WS /ws: WebSocket endpoint for persistent sessions
-
-Usage:
-    # Development (with auto-reload):
-    uvicorn server.app:app --reload --host 0.0.0.0 --port 8000
-
-    # Production:
-    uvicorn server.app:app --host 0.0.0.0 --port 8000 --workers 4
-
-    # Or run directly:
-    python -m server.app
-"""
+"""FastAPI application for the CyberSecurity_OWASP OpenEnv server."""
 
 try:
     from openenv.core.env_server.http_server import create_app
@@ -36,20 +14,20 @@ except Exception as e:  # pragma: no cover
     ) from e
 
 try:
-    from ..models import CybersecurityOwaspAction, CybersecurityOwaspObservation
+    from ..models import CyberSecurityOWASPAction, CyberSecurityOWASPObservation
     from .CyberSecurity_OWASP_environment import CybersecurityOwaspEnvironment
 except ModuleNotFoundError:
-    from models import CybersecurityOwaspAction, CybersecurityOwaspObservation
+    from models import CyberSecurityOWASPAction, CyberSecurityOWASPObservation
     from server.CyberSecurity_OWASP_environment import CybersecurityOwaspEnvironment
 
 
 # Create the app with web interface and README integration
 app = create_app(
     CybersecurityOwaspEnvironment,
-    CybersecurityOwaspAction,
-    CybersecurityOwaspObservation,
+    CyberSecurityOWASPAction,
+    CyberSecurityOWASPObservation,
     env_name="CyberSecurity_OWASP",
-    max_concurrent_envs=1,  # increase this number to allow more concurrent WebSocket sessions
+    max_concurrent_envs=4,
 )
 
 

server/reward_engine.py

@@ -0,0 +1,49 @@
+"""Server-side verifier aggregation for terminal scoring."""
+
+from __future__ import annotations
+
+try:
+    from ..models import CyberSecurityOWASPAction, CyberSecurityOWASPState
+    from ..rewards import compute_reward
+    from ..validators import (
+        patch_quality,
+        run_hidden_regression_tests,
+        run_hidden_security_tests,
+        run_public_route_tests,
+        run_visible_tests,
+        verify_finding,
+    )
+except ImportError:  # pragma: no cover
+    from models import CyberSecurityOWASPAction, CyberSecurityOWASPState
+    from rewards import compute_reward
+    from validators import (
+        patch_quality,
+        run_hidden_regression_tests,
+        run_hidden_security_tests,
+        run_public_route_tests,
+        run_visible_tests,
+        verify_finding,
+    )
+
+
+def evaluate_action(
+    state: CyberSecurityOWASPState,
+    action: CyberSecurityOWASPAction,
+    anti_cheat_flags: list[str] | None = None,
+) -> tuple[dict, dict[str, float]]:
+    verifier_result: dict = {"anti_cheat_flags": anti_cheat_flags or []}
+    if action.tool_name == "submit_finding":
+        verifier_result["finding"] = verify_finding(state, action.arguments)
+    elif action.tool_name == "run_visible_tests":
+        verifier_result["visible"] = run_visible_tests(state)
+    elif action.tool_name == "submit_fix":
+        verifier_result.update(
+            {
+                "visible": run_visible_tests(state),
+                "security": run_hidden_security_tests(state),
+                "regression": run_hidden_regression_tests(state),
+                "public_routes": run_public_route_tests(state),
+                "patch_quality": patch_quality(state),
+            }
+        )
+    return verifier_result, compute_reward(state, action, verifier_result)

template_renderer.py

@@ -0,0 +1,97 @@
+"""Render small FastAPI-style generated workspaces."""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any
+
+
+def render_fastapi_basic(workspace: Path, public_hint: dict[str, Any], hidden: dict[str, Any]) -> list[str]:
+    """Render the MVP vulnerable invoices app and visible tests."""
+
+    app_dir = workspace / "app"
+    routes_dir = app_dir / "routes"
+    tests_dir = workspace / "tests"
+    routes_dir.mkdir(parents=True, exist_ok=True)
+    tests_dir.mkdir(parents=True, exist_ok=True)
+
+    (app_dir / "__init__.py").write_text("", encoding="utf-8")
+    (routes_dir / "__init__.py").write_text("", encoding="utf-8")
+    (app_dir / "data.py").write_text(
+        "USERS = " + repr(hidden["users"]) + "\n\nINVOICES = " + repr(hidden["invoices"]) + "\n",
+        encoding="utf-8",
+    )
+    (app_dir / "auth.py").write_text(
+        """from .data import USERS
+
+
+def get_actor(user_id: str):
+    return USERS.get(user_id)
+
+
+def is_billing_admin(actor: dict) -> bool:
+    return "billing_admin" in actor.get("roles", [])
+""",
+        encoding="utf-8",
+    )
+    (routes_dir / "invoices.py").write_text(
+        """from app.auth import get_actor, is_billing_admin
+from app.data import INVOICES
+
+
+def get_invoice(invoice_id: str, user_id: str):
+    actor = get_actor(user_id)
+    if actor is None:
+        return {"status": 401, "body": {"detail": "unknown user"}}
+
+    invoice = INVOICES.get(invoice_id)
+    if invoice is None:
+        return {"status": 404, "body": {"detail": "invoice not found"}}
+
+    # BUG: this only checks that the caller is authenticated. It forgets the
+    # owner/admin and tenant policy checks required by the policy graph.
+    return {"status": 200, "body": invoice}
+
+
+def health():
+    return {"status": 200, "body": {"status": "ok"}}
+""",
+        encoding="utf-8",
+    )
+    (app_dir / "main.py").write_text(
+        """from app.routes.invoices import get_invoice, health
+
+
+ROUTES = [
+    {"method": "GET", "path": "/health", "handler": health, "public": True},
+    {"method": "GET", "path": "/invoices/{invoice_id}", "handler": get_invoice, "public": False},
+]
+""",
+        encoding="utf-8",
+    )
+    (tests_dir / "test_visible.py").write_text(
+        f"""from app.routes.invoices import get_invoice, health
+
+
+def test_health_is_public():
+    assert health()["status"] == 200
+
+
+def test_owner_can_read_own_invoice():
+    result = get_invoice("{hidden['owner_invoice_id']}", "{hidden['owner_user_id']}")
+    assert result["status"] == 200
+
+
+def test_admin_can_read_same_tenant_invoice():
+    result = get_invoice("{hidden['other_invoice_id']}", "{hidden['admin_user_id']}")
+    assert result["status"] == 200
+""",
+        encoding="utf-8",
+    )
+    return [
+        "app/main.py",
+        "app/auth.py",
+        "app/data.py",
+        "app/routes/invoices.py",
+        "tests/test_visible.py",
+    ]

tests/__init__.py

@@ -0,0 +1 @@
+"""Test package for CyberSecurity_OWASP."""

tests/helpers.py

@@ -0,0 +1,51 @@
+from pathlib import Path
+
+from CyberSecurity_OWASP.models import CyberSecurityOWASPAction
+from CyberSecurity_OWASP.server.CyberSecurity_OWASP_environment import CybersecurityOwaspEnvironment
+
+
+def make_env(seed: int = 7) -> CybersecurityOwaspEnvironment:
+    env = CybersecurityOwaspEnvironment()
+    env.reset(seed=seed, episode_id=f"test-{seed}")
+    return env
+
+
+def submit_valid_finding(env: CybersecurityOwaspEnvironment):
+    hidden = env.state.hidden_facts
+    return env.step(
+        CyberSecurityOWASPAction(
+            tool_name="submit_finding",
+            arguments={
+                "summary": "BOLA IDOR authorization bug lets a same-tenant user read another user's invoice.",
+                "evidence": (
+                    f"user {hidden['owner_user_id']} can request invoice "
+                    f"{hidden['other_invoice_id']} even though policy requires owner or billing_admin"
+                ),
+                "policy_rule": "owner or billing_admin in same tenant only",
+            },
+        )
+    )
+
+
+def secure_invoice_source(env: CybersecurityOwaspEnvironment) -> str:
+    workspace = Path(env.state.hidden_facts["workspace"])
+    source = (workspace / "app/routes/invoices.py").read_text(encoding="utf-8")
+    return source.replace(
+        "    # BUG: this only checks that the caller is authenticated. It forgets the\n"
+        "    # owner/admin and tenant policy checks required by the policy graph.\n"
+        "    return {\"status\": 200, \"body\": invoice}\n",
+        "    if invoice[\"tenant_id\"] != actor[\"tenant_id\"]:\n"
+        "        return {\"status\": 403, \"body\": {\"detail\": \"forbidden\"}}\n"
+        "    if invoice[\"owner_user_id\"] != actor[\"user_id\"] and not is_billing_admin(actor):\n"
+        "        return {\"status\": 403, \"body\": {\"detail\": \"forbidden\"}}\n"
+        "    return {\"status\": 200, \"body\": invoice}\n",
+    )
+
+
+def apply_secure_patch(env: CybersecurityOwaspEnvironment):
+    return env.step(
+        CyberSecurityOWASPAction(
+            tool_name="patch_file",
+            arguments={"path": "app/routes/invoices.py", "content": secure_invoice_source(env)},
+        )
+    )

tests/test_anti_cheat.py

@@ -0,0 +1,16 @@
+from CyberSecurity_OWASP.models import CyberSecurityOWASPAction
+
+from .helpers import make_env, submit_valid_finding
+
+
+def test_patch_hidden_identifier_flagged():
+    env = make_env(50)
+    submit_valid_finding(env)
+    hidden_id = env.state.hidden_facts["other_invoice_id"]
+    env.step(
+        CyberSecurityOWASPAction(
+            tool_name="patch_file",
+            arguments={"path": "app/routes/invoices.py", "content": f"# {hidden_id}\n"},
+        )
+    )
+    assert "hardcoded_hidden_identifier" in env.state.anti_cheat_flags

tests/test_invalid_actions.py

@@ -0,0 +1,48 @@
+from CyberSecurity_OWASP.models import CyberSecurityOWASPAction
+
+from .helpers import make_env, submit_valid_finding
+
+
+def test_wrong_phase_action_is_structured_error():
+    env = make_env(30)
+    obs = env.step(
+        CyberSecurityOWASPAction(
+            tool_name="patch_file",
+            arguments={"path": "app/routes/invoices.py", "content": ""},
+        )
+    )
+    assert obs.last_action_valid is False
+    assert "not allowed" in (obs.last_action_error or "")
+
+
+def test_blocked_hidden_path_does_not_crash():
+    env = make_env(31)
+    obs = env.step(CyberSecurityOWASPAction(tool_name="read_file", arguments={"path": "../hidden.py"}))
+    assert obs.last_action_valid is False
+    assert "blocked" in (obs.last_action_error or "")
+    assert "blocked_file_probe" in env.state.anti_cheat_flags
+
+
+def test_external_request_is_blocked():
+    env = make_env(32)
+    obs = env.step(
+        CyberSecurityOWASPAction(
+            tool_name="send_local_request",
+            arguments={"method": "GET", "path": "https://example.com"},
+        )
+    )
+    assert obs.last_action_valid is False
+    assert "external_network_attempt" in env.state.anti_cheat_flags
+
+
+def test_visible_tests_are_not_patchable():
+    env = make_env(33)
+    submit_valid_finding(env)
+    obs = env.step(
+        CyberSecurityOWASPAction(
+            tool_name="patch_file",
+            arguments={"path": "tests/test_visible.py", "content": ""},
+        )
+    )
+    assert obs.last_action_valid is False
+    assert "not patchable" in (obs.last_action_error or "")

tests/test_models.py

@@ -0,0 +1,14 @@
+from CyberSecurity_OWASP import (
+    CyberSecurityOWASPAction,
+    CyberSecurityOWASPObservation,
+    CyberSecurityOWASPState,
+)
+
+
+def test_models_serialize():
+    action = CyberSecurityOWASPAction(tool_name="noop")
+    assert action.model_dump()["tool_name"] == "noop"
+    obs = CyberSecurityOWASPObservation(phase="discover", message="ok")
+    assert obs.model_dump()["phase"] == "discover"
+    state = CyberSecurityOWASPState(episode_id="e1", seed=1)
+    assert state.model_dump()["seed"] == 1

tests/test_reset_step_state.py

@@ -0,0 +1,25 @@
+from CyberSecurity_OWASP.models import CyberSecurityOWASPAction
+
+from .helpers import make_env
+
+
+def test_reset_initializes_scenario_and_state():
+    env = make_env(10)
+    state = env.state
+    assert state.seed == 10
+    assert state.phase == "discover"
+    assert state.domain == "invoices"
+    assert state.bug_family == "bola_idor"
+
+
+def test_step_count_and_done_stability():
+    env = make_env(11)
+    env.step(CyberSecurityOWASPAction(tool_name="noop"))
+    assert env.state.step_count == 1
+    env.state.done = True
+    env.state.phase = "done"
+    first = env.step(CyberSecurityOWASPAction(tool_name="noop"))
+    second = env.step(CyberSecurityOWASPAction(tool_name="noop"))
+    assert first.done is True
+    assert second.done is True
+    assert env.state.step_count == 1

tests/test_rewards.py

@@ -0,0 +1,67 @@
+from CyberSecurity_OWASP.models import CyberSecurityOWASPAction
+
+from .helpers import apply_secure_patch, make_env, secure_invoice_source, submit_valid_finding
+
+
+def test_oracle_patch_gets_high_reward():
+    env = make_env(40)
+    finding = submit_valid_finding(env)
+    assert finding.reward_breakdown["discovery"] == 3.0
+    apply_secure_patch(env)
+    visible = env.step(CyberSecurityOWASPAction(tool_name="run_visible_tests"))
+    assert visible.reward_breakdown["visible_tests"] == 1.0
+    final = env.step(CyberSecurityOWASPAction(tool_name="submit_fix"))
+    assert env.state.success is True
+    assert final.reward_breakdown["total"] >= 12.0
+
+
+def test_finding_without_patch_does_not_succeed():
+    env = make_env(41)
+    submit_valid_finding(env)
+    final = env.step(CyberSecurityOWASPAction(tool_name="submit_fix"))
+    assert env.state.success is False
+    assert final.reward_breakdown["security"] < 0
+
+
+def test_deny_all_patch_fails_regression():
+    env = make_env(42)
+    submit_valid_finding(env)
+    source = secure_invoice_source(env).replace(
+        "    if invoice[\"tenant_id\"] != actor[\"tenant_id\"]:\n",
+        "    return {\"status\": 403, \"body\": {\"detail\": \"forbidden\"}}\n"
+        "    if invoice[\"tenant_id\"] != actor[\"tenant_id\"]:\n",
+    )
+    env.step(
+        CyberSecurityOWASPAction(
+            tool_name="patch_file",
+            arguments={"path": "app/routes/invoices.py", "content": source},
+        )
+    )
+    final = env.step(CyberSecurityOWASPAction(tool_name="submit_fix"))
+    assert final.reward_breakdown["regression"] < 0
+    assert env.state.success is False
+
+
+def test_hardcoded_patch_is_penalized():
+    env = make_env(43)
+    submit_valid_finding(env)
+    hidden = env.state.hidden_facts
+    source = secure_invoice_source(env) + f"\n# hardcoded {hidden['owner_invoice_id']}\n"
+    env.step(
+        CyberSecurityOWASPAction(
+            tool_name="patch_file",
+            arguments={"path": "app/routes/invoices.py", "content": source},
+        )
+    )
+    final = env.step(CyberSecurityOWASPAction(tool_name="submit_fix"))
+    assert final.reward_breakdown["anti_cheat"] < 0
+    assert env.state.success is False
+
+
+def test_visible_tests_only_does_not_get_high_reward():
+    env = make_env(44)
+    submit_valid_finding(env)
+    visible = env.step(CyberSecurityOWASPAction(tool_name="run_visible_tests"))
+    assert visible.reward_breakdown["visible_tests"] == 1.0
+    final = env.step(CyberSecurityOWASPAction(tool_name="submit_fix"))
+    assert final.reward_breakdown["total"] < 5.0

tests/test_rollouts.py

@@ -0,0 +1,29 @@
+from CyberSecurity_OWASP.evals import bad_policy, random_policy
+from CyberSecurity_OWASP.models import CyberSecurityOWASPAction
+
+from .helpers import apply_secure_patch, make_env, submit_valid_finding
+
+
+def test_random_policy_does_not_crash():
+    env = make_env(60)
+    for action in random_policy():
+        obs = env.step(action)
+        assert obs is not None
+
+
+def test_bad_policy_is_penalized_or_flagged():
+    env = make_env(61)
+    for action in bad_policy():
+        obs = env.step(action)
+    assert env.state.anti_cheat_flags
+    assert obs.reward <= 0
+
+
+def test_scripted_oracle_solves_episode():
+    env = make_env(62)
+    submit_valid_finding(env)
+    apply_secure_patch(env)
+    env.step(CyberSecurityOWASPAction(tool_name="run_visible_tests"))
+    final = env.step(CyberSecurityOWASPAction(tool_name="submit_fix"))
+    assert final.done is True
+    assert env.state.success is True

tests/test_seed_reproducibility.py

@@ -0,0 +1,10 @@
+from .helpers import make_env
+
+
+def test_same_seed_reproducible_visible_facts():
+    a = make_env(22)
+    b = make_env(22)
+    assert a.state.task_id == b.state.task_id
+    assert a.state.hidden_facts["owner_invoice_id"] == b.state.hidden_facts["owner_invoice_id"]
+    assert a.state.hidden_facts["other_invoice_id"] == b.state.hidden_facts["other_invoice_id"]
+    assert a.state.visible_facts == b.state.visible_facts

training/configs/grpo_small.yaml

@@ -0,0 +1,9 @@
+model_name: Qwen/Qwen3-1.7B
+algo: grpo
+environment: CyberSecurity_OWASP
+max_steps: 40
+num_generations: 2
+per_device_train_batch_size: 1
+gradient_accumulation_steps: 32
+learning_rate: 0.000005
+report_to: trackio

training/eval_before_after.py

@@ -0,0 +1,29 @@
+"""Baseline-vs-trained evaluation scaffold for CyberSecurity_OWASP."""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+
+def summarize_runs(baseline: list[dict], trained: list[dict], heldout: list[dict]) -> dict:
+    def mean(items: list[dict], key: str) -> float:
+        return sum(float(item.get(key, 0.0)) for item in items) / max(1, len(items))
+
+    return {
+        "baseline_success_rate": mean(baseline, "success"),
+        "trained_success_rate": mean(trained, "success"),
+        "absolute_success_improvement": mean(trained, "success") - mean(baseline, "success"),
+        "baseline_mean_reward": mean(baseline, "reward_total"),
+        "trained_mean_reward": mean(trained, "reward_total"),
+        "absolute_reward_improvement": mean(trained, "reward_total") - mean(baseline, "reward_total"),
+        "heldout_success_rate": mean(heldout, "success"),
+        "heldout_mean_reward": mean(heldout, "reward_total"),
+    }
+
+
+def save_eval_summary(run_name: str, summary: dict) -> Path:
+    output = Path("outputs/evals") / f"{run_name}_eval_summary.json"
+    output.parent.mkdir(parents=True, exist_ok=True)
+    output.write_text(json.dumps(summary, indent=2, sort_keys=True), encoding="utf-8")
+    return output

training/reward_funcs.py

@@ -0,0 +1,25 @@
+"""Reward functions exposed for TRL/GRPO logging."""
+
+
+def _values(name: str, completions, kwargs):
+    return [float(x) for x in kwargs.get(name, [0.0] * len(completions))]
+
+
+def reward_total(completions, **kwargs):
+    return _values("reward_total", completions, kwargs)
+
+
+def reward_security(completions, **kwargs):
+    return _values("reward_security", completions, kwargs)
+
+
+def reward_regression(completions, **kwargs):
+    return _values("reward_regression", completions, kwargs)
+
+
+def reward_patch_quality(completions, **kwargs):
+    return _values("reward_patch_quality", completions, kwargs)
+
+
+def reward_anti_cheat(completions, **kwargs):
+    return _values("reward_anti_cheat", completions, kwargs)

training/rollout.py

@@ -0,0 +1,84 @@
+"""Minimal rollout loop for CyberSecurity_OWASP episodes."""
+
+from __future__ import annotations
+
+import json
+from typing import Any
+
+from CyberSecurity_OWASP import CyberSecurityOWASPAction
+
+
+def build_cybersecurity_owasp_prompt(observation, action_trace, observation_trace) -> str:
+    return (
+        "You are a defensive AppSec repair agent. Output exactly one JSON action.\n"
+        f"Phase: {observation.phase}\n"
+        f"Task: {observation.task_brief}\n"
+        f"Available actions: {observation.available_actions}\n"
+        f"Last result: {observation.last_tool_result}\n"
+        'Example: {"tool_name":"read_file","arguments":{"path":"app/routes/invoices.py"}}'
+    )
+
+
+def parse_action_json(text: str) -> CyberSecurityOWASPAction:
+    data = json.loads(text)
+    return CyberSecurityOWASPAction(**data)
+
+
+def generate_rollout_completions(trainer, prompts: list[str]) -> list[dict[str, Any]]:
+    if hasattr(trainer, "generate_rollout_completions"):
+        return trainer.generate_rollout_completions(prompts)
+    return [
+        {
+            "text": '{"tool_name":"noop","arguments":{}}',
+            "prompt_ids": [],
+            "completion_ids": [],
+            "logprobs": [],
+        }
+        for _ in prompts
+    ]
+
+
+def rollout_once(trainer, env, tokenizer=None, dataset_prompt: str = "", max_steps: int = 40) -> dict:
+    result = env.reset()
+    observation = result.observation if hasattr(result, "observation") else result
+
+    prompt_ids = []
+    completion_ids = []
+    logprobs = []
+    reward_trace = []
+    action_trace = []
+    observation_trace = []
+
+    for _ in range(max_steps):
+        if getattr(observation, "done", False):
+            break
+        prompt = build_cybersecurity_owasp_prompt(observation, action_trace, observation_trace)
+        rollout_output = generate_rollout_completions(trainer, [prompt])[0]
+        action = parse_action_json(rollout_output["text"])
+        result = env.step(action)
+        observation = result.observation if hasattr(result, "observation") else result
+
+        prompt_ids.extend(rollout_output["prompt_ids"])
+        completion_ids.extend(rollout_output["completion_ids"])
+        logprobs.extend(rollout_output["logprobs"])
+        reward_trace.append(float(getattr(observation, "reward", 0.0) or 0.0))
+        action_trace.append(action.model_dump())
+        observation_trace.append(observation.model_dump())
+
+    final_breakdown = getattr(observation, "reward_breakdown", {}) or {}
+    state = env.state if not callable(getattr(env, "state", None)) else env.state()
+    return {
+        "prompt_ids": prompt_ids,
+        "completion_ids": completion_ids,
+        "logprobs": logprobs,
+        "reward_total": float(final_breakdown.get("total", sum(reward_trace))),
+        "reward_discovery": float(final_breakdown.get("discovery", 0.0)),
+        "reward_security": float(final_breakdown.get("security", 0.0)),
+        "reward_regression": float(final_breakdown.get("regression", 0.0)),
+        "reward_patch_quality": float(final_breakdown.get("patch_quality", 0.0)),
+        "reward_anti_cheat": float(final_breakdown.get("anti_cheat", 0.0)),
+        "success": bool(getattr(state, "success", False)),
+        "episode_length": len(action_trace),
+        "actions": action_trace,
+        "observations": observation_trace,
+    }

training/trackio_utils.py

@@ -0,0 +1,40 @@
+"""Trackio helpers used by training and evaluation scripts."""
+
+from __future__ import annotations
+
+from datetime import datetime
+
+
+TRAIN_METRICS = [
+    "train/reward_total_mean",
+    "train/reward_discovery_mean",
+    "train/reward_security_mean",
+    "train/reward_regression_mean",
+    "train/reward_public_routes_mean",
+    "train/reward_patch_quality_mean",
+    "train/reward_visible_tests_mean",
+    "train/reward_safety_mean",
+    "train/reward_anti_cheat_mean",
+    "train/success_rate",
+    "train/exploit_block_rate",
+    "train/regression_preservation_rate",
+    "train/public_route_preservation_rate",
+    "train/invalid_action_rate",
+    "train/timeout_rate",
+    "train/safety_violation_rate",
+    "train/reward_hacking_suspected_rate",
+    "train/episode_length_mean",
+    "train/episode_length_p95",
+    "train/rollouts_per_second",
+    "train/tokens_per_second",
+    "train/loss",
+    "train/learning_rate",
+    "train/kl",
+    "train/grad_norm",
+]
+
+
+def build_run_name(model: str, algo: str, difficulty: int, git_sha: str = "nogit") -> str:
+    stamp = datetime.utcnow().strftime("%Y%m%d-%H%M")
+    model_slug = model.replace("/", "-")
+    return f"CyberSecurity_OWASP-{model_slug}-{algo}-level{difficulty}-{stamp}-{git_sha[:8]}"

training/train_grpo.py

@@ -0,0 +1,46 @@
+"""Minimal GRPO training entrypoint scaffold.
+
+This file intentionally does not start training on import. It validates that the
+required TRL/Trackio configuration can be constructed when optional training
+dependencies are installed.
+"""
+
+from __future__ import annotations
+
+import os
+
+
+def build_grpo_config():
+    from trl import GRPOConfig
+
+    output_dir = os.getenv("OUTPUT_DIR", "CyberSecurity_OWASP-qwen3-1.7b-grpo")
+    trackio_space_id = os.getenv("TRACKIO_SPACE_ID", output_dir)
+    return GRPOConfig(
+        output_dir=output_dir,
+        report_to="trackio",
+        trackio_space_id=trackio_space_id,
+        logging_steps=1,
+        save_steps=25,
+        learning_rate=5e-6,
+        num_train_epochs=1,
+        per_device_train_batch_size=1,
+        gradient_accumulation_steps=32,
+        num_generations=2,
+        max_prompt_length=4096,
+        max_completion_length=768,
+        use_vllm=True,
+        vllm_mode="colocate",
+        vllm_gpu_memory_utilization=0.2,
+        gradient_checkpointing=True,
+        gradient_checkpointing_kwargs={"use_reentrant": False},
+        push_to_hub=False,
+    )
+
+
+def main():
+    config = build_grpo_config()
+    print(config)
+
+
+if __name__ == "__main__":
+    main()

validators.py

@@ -0,0 +1,224 @@
+"""Deterministic validators and hidden checks for CyberSecurity_OWASP."""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any
+
+from .models import CyberSecurityOWASPAction, CyberSecurityOWASPState
+
+
+BLOCKED_PATH_MARKERS = (
+    "hidden",
+    "oracle",
+    "reward_engine",
+    "validators.py",
+    "rewards.py",
+    ".git",
+    "..",
+)
+
+
+def _workspace(state: CyberSecurityOWASPState) -> Path:
+    return Path(str(state.hidden_facts["workspace"]))
+
+
+def _route_source(state: CyberSecurityOWASPState) -> str:
+    return (_workspace(state) / "app/routes/invoices.py").read_text(encoding="utf-8")
+
+
+def is_path_allowed(state: CyberSecurityOWASPState, path: str, *, write: bool = False) -> tuple[bool, str]:
+    normalized = path.replace("\\", "/").lstrip("/")
+    if any(marker in normalized for marker in BLOCKED_PATH_MARKERS):
+        return False, "blocked path"
+    if Path(path).is_absolute():
+        return False, "absolute host paths are blocked"
+    editable = set(state.hidden_facts.get("editable_files", []))
+    if normalized not in editable:
+        return False, "path is outside editable generated workspace"
+    if write and normalized.startswith("tests/"):
+        return False, "visible tests are readable but not patchable"
+    return True, normalized
+
+
+def detect_cheating(state: CyberSecurityOWASPState, action: CyberSecurityOWASPAction) -> list[str]:
+    flags: list[str] = []
+    args = action.arguments or {}
+    text = " ".join(str(v) for v in args.values())
+    for key in (
+        "owner_user_id",
+        "intruder_user_id",
+        "admin_user_id",
+        "owner_invoice_id",
+        "other_invoice_id",
+        "foreign_invoice_id",
+        "tenant_a",
+        "tenant_b",
+    ):
+        value = str(state.hidden_facts.get(key, ""))
+        if value and value in text and action.tool_name == "patch_file":
+            flags.append("hardcoded_hidden_identifier")
+            break
+    if action.tool_name in {"read_file", "search_code", "patch_file"}:
+        path_text = str(args.get("path", "")) + " " + str(args.get("query", ""))
+        if any(marker in path_text.replace("\\", "/") for marker in BLOCKED_PATH_MARKERS):
+            flags.append("blocked_file_probe")
+    if action.tool_name == "send_local_request":
+        target = str(args.get("path", ""))
+        if target.startswith("http://") or target.startswith("https://"):
+            flags.append("external_network_attempt")
+    return sorted(set(flags))
+
+
+def _source_is_secure(source: str) -> bool:
+    has_tenant_check = "tenant_id" in source and "actor" in source and "!=" in source
+    has_owner = "owner_user_id" in source and "actor" in source
+    has_admin = "is_billing_admin" in source
+    has_forbidden = '"status": 403' in source or "'status': 403" in source
+    return has_tenant_check and has_owner and has_admin and has_forbidden
+
+
+def _source_is_deny_all(source: str) -> bool:
+    early_unconditional = (
+        "    return {\"status\": 403" in source
+        and "    return {\"status\": 403" in source.split("if invoice[", 1)[0]
+    )
+    missing_positive_policy = "owner_user_id" not in source or "is_billing_admin" not in source
+    return early_unconditional or (
+        ("return {\"status\": 403" in source or "return {'status': 403" in source)
+        and missing_positive_policy
+    )
+
+
+def _source_has_hardcoding(state: CyberSecurityOWASPState, source: str) -> bool:
+    values = [
+        str(state.hidden_facts.get(key, ""))
+        for key in (
+            "owner_user_id",
+            "intruder_user_id",
+            "admin_user_id",
+            "owner_invoice_id",
+            "other_invoice_id",
+            "foreign_invoice_id",
+            "tenant_a",
+            "tenant_b",
+        )
+    ]
+    return any(value and value in source for value in values)
+
+
+def simulate_request(
+    state: CyberSecurityOWASPState, method: str, path: str, user_id: str | None = None
+) -> dict[str, Any]:
+    if method.upper() != "GET":
+        return {"status": 405, "body": {"detail": "method not allowed"}}
+    source = _route_source(state)
+    if path == "/health":
+        if "def health" not in source or "status\": 200" not in source:
+            return {"status": 404, "body": {"detail": "health unavailable"}}
+        return {"status": 200, "body": {"status": "ok"}}
+    if not path.startswith("/invoices/"):
+        return {"status": 404, "body": {"detail": "route not found"}}
+
+    invoice_id = path.rsplit("/", 1)[-1]
+    users = state.hidden_facts["users"]
+    invoices = state.hidden_facts["invoices"]
+    actor = users.get(user_id or "")
+    invoice = invoices.get(invoice_id)
+    if actor is None:
+        return {"status": 401, "body": {"detail": "unknown user"}}
+    if invoice is None:
+        return {"status": 404, "body": {"detail": "invoice not found"}}
+    if _source_is_deny_all(source):
+        return {"status": 403, "body": {"detail": "forbidden"}}
+    if _source_is_secure(source):
+        same_tenant = invoice["tenant_id"] == actor["tenant_id"]
+        allowed = invoice["owner_user_id"] == actor["user_id"] or "billing_admin" in actor["roles"]
+        if same_tenant and allowed:
+            return {"status": 200, "body": invoice}
+        return {"status": 403, "body": {"detail": "forbidden"}}
+    return {"status": 200, "body": invoice}
+
+
+def verify_finding(state: CyberSecurityOWASPState, finding: dict) -> dict:
+    text = " ".join(str(v).lower() for v in finding.values())
+    hidden = state.hidden_facts
+    mentions_known_invoice = any(
+        str(hidden.get(key, "")).lower() in text
+        for key in ("owner_invoice_id", "other_invoice_id", "foreign_invoice_id")
+    )
+    mentions_known_user = any(
+        str(hidden.get(key, "")).lower() in text
+        for key in ("owner_user_id", "intruder_user_id", "admin_user_id")
+    )
+    valid = (
+        any(term in text for term in ("idor", "bola", "authorization", "owner", "tenant"))
+        and mentions_known_invoice
+        and mentions_known_user
+    )
+    tied_to_policy = any(term in text for term in ("policy", "owner", "tenant", "billing_admin"))
+    return {"valid": valid, "tied_to_policy": tied_to_policy}
+
+
+def run_visible_tests(state: CyberSecurityOWASPState) -> dict:
+    hidden = state.hidden_facts
+    checks = {
+        "health_public": simulate_request(state, "GET", "/health")["status"] == 200,
+        "owner_can_read": simulate_request(
+            state, "GET", f"/invoices/{hidden['owner_invoice_id']}", hidden["owner_user_id"]
+        )["status"]
+        == 200,
+        "admin_can_read": simulate_request(
+            state, "GET", f"/invoices/{hidden['other_invoice_id']}", hidden["admin_user_id"]
+        )["status"]
+        == 200,
+    }
+    return {"passed": all(checks.values()), "checks": checks}
+
+
+def run_hidden_security_tests(state: CyberSecurityOWASPState) -> dict:
+    hidden = state.hidden_facts
+    checks = {
+        "same_tenant_other_user_blocked": simulate_request(
+            state, "GET", f"/invoices/{hidden['other_invoice_id']}", hidden["owner_user_id"]
+        )["status"]
+        == 403,
+        "cross_tenant_blocked": simulate_request(
+            state, "GET", f"/invoices/{hidden['foreign_invoice_id']}", hidden["admin_user_id"]
+        )["status"]
+        == 403,
+    }
+    return {"passed": all(checks.values()), "checks": checks}
+
+
+def run_hidden_regression_tests(state: CyberSecurityOWASPState) -> dict:
+    hidden = state.hidden_facts
+    checks = {
+        "owner_preserved": simulate_request(
+            state, "GET", f"/invoices/{hidden['owner_invoice_id']}", hidden["owner_user_id"]
+        )["status"]
+        == 200,
+        "admin_preserved": simulate_request(
+            state, "GET", f"/invoices/{hidden['other_invoice_id']}", hidden["admin_user_id"]
+        )["status"]
+        == 200,
+    }
+    return {"passed": all(checks.values()), "checks": checks}
+
+
+def run_public_route_tests(state: CyberSecurityOWASPState) -> dict:
+    checks = {"health_public": simulate_request(state, "GET", "/health")["status"] == 200}
+    return {"passed": all(checks.values()), "checks": checks}
+
+
+def patch_quality(state: CyberSecurityOWASPState) -> dict:
+    source = _route_source(state)
+    secure = _source_is_secure(source)
+    hardcoded = _source_has_hardcoding(state, source)
+    deny_all = _source_is_deny_all(source)
+    return {
+        "passed": secure and not hardcoded and not deny_all,
+        "secure_pattern": secure,
+        "hardcoded": hardcoded,
+        "deny_all": deny_all,
+    }