AGENTS.md

AGENTS.md — CyberSecurity_OWASP Builder Instructions

Purpose

This repository implements CyberSecurity_OWASP, an OpenEnv-compliant RL environment for training a single LLM agent to perform a defensive application-security workflow:

inspect generated app + policy -> discover authorization bug -> submit safe finding -> patch code -> preserve intended behavior

The environment must train the model to do real interactive work, not answer static security questions. The model must act step by step through typed OpenEnv actions, observe consequences, receive deterministic reward, and improve through RL.

The canonical repository and OpenEnv environment name is CyberSecurity_OWASP. Use this exact name in openenv.yaml, pyproject.toml, HF Spaces repo naming, Docker image tags, Trackio run names, command examples, and documentation.

The target stack is:

CyberSecurity_OWASP OpenEnv environment
  -> deterministic verifier + hidden tests
  -> rollout loop
  -> HF TRL / Unsloth GRPO
  -> Trackio logging
  -> held-out evaluation
  -> HF Spaces deployment

The final project must show measurable improvement in reward, exploit-block rate, regression-preservation rate, and held-out generalization after training.

---

Product definition

CyberSecurity_OWASP generates a new local application scenario every reset(seed). Each episode contains:

• a policy graph describing users, roles, tenants, resources, ownership, permissions, and public routes;
• a generated FastAPI-style application workspace;
• exactly one injected OWASP A01-style authorization defect;
• visible tests for normal behavior;
• hidden invariant tests for authorization correctness, regression protection, public-route preservation, and anti-cheat checks.

The environment has one LLM agent, not separate red-team and blue-team LLMs. The environment itself acts as the scenario generator, tool server, verifier, and judge.

---

Highest-priority objectives

When making implementation decisions, optimize in this order:

1. Verifier correctness: deterministic tests must decide whether the patch actually fixes the authorization defect.
2. Reward integrity: reward must be hard to hack and must punish insecure or regressive patches.
3. Anti-overfitting: the model must generalize across apps, layouts, policies, domains, names, and bug families.
4. OpenEnv compliance: expose typed Action, Observation, and State; implement reset(), step(action), and state correctly.
5. Trainability: baseline model should sometimes get partial reward; curriculum should make early learning possible.
6. Real-world usefulness: the workflow should resemble secure code review / AppSec authorization repair.
7. Demo clarity: show before/after rollouts, reward curves, and why the trained model improved.
8. Hackathon competitiveness: prioritize a novel, interactive, professionally useful environment with a coherent training pipeline.

Do not train before the environment, verifier, anti-cheat tests, and before/after evaluation are stable.

---

Hackathon alignment requirements

The implementation must satisfy these minimum requirements:

• use the latest OpenEnv release;
• include a minimal HF TRL or Unsloth training script;
• use Trackio as the default tracker for training and evaluation;
• be deployable as an OpenEnv-compliant Hugging Face Space;
• include a README / mini-blog style explanation;
• show baseline-vs-trained improvement.

Optimize for judging:

Criterion	Weight	CyberSecurity_OWASP evidence
Environment innovation	40%	procedural OWASP authorization-repair environment with generated code, policy, and hidden verifier
Storytelling	30%	single LLM learns discover + patch, before/after security behavior
Showing improvement in rewards	20%	reward curves, exploit-block pass rate, regression-preservation rate
Reward/training pipeline	10%	deterministic reward, GRPO/PPO rollout loop, Trackio metrics

---

Non-negotiable environment design

CyberSecurity_OWASP must be a single-agent environment:

phase = discover -> patch -> done

Do not implement a two-LLM red-team/blue-team setup. The single model must learn both discovery and repair.

The environment must be defensive and local only. It must never target real systems or teach unauthorized exploitation. All probing must be limited to the generated local workspace controlled by the environment.

---

Required repository structure

Prefer this structure:

.
├── AGENTS.md
├── README.md
├── 00_PROJECT_BRIEF.md
├── 01_ARCHITECTURE.md
├── pyproject.toml
├── openenv.yaml
├── envs/
│   └── CyberSecurity_OWASP/
│       ├── __init__.py
│       ├── models.py
│       ├── client.py
│       ├── README.md
│       ├── rewards.py
│       ├── validators.py
│       ├── safety.py
│       ├── evals.py
│       ├── server/
│       │   ├── __init__.py
│       │   ├── app.py
│       │   ├── environment.py
│       │   ├── scenario_compiler.py
│       │   ├── policy_graph.py
│       │   ├── template_renderer.py
│       │   ├── bug_mutator.py
│       │   ├── fixture_generator.py
│       │   ├── reward_engine.py
│       │   ├── requirements.txt
│       │   └── Dockerfile
│       ├── templates/
│       │   └── fastapi_basic/
│       ├── scenario_cache/
│       │   ├── train/
│       │   ├── validation/
│       │   └── hidden_eval/
│       └── tests/
│           ├── test_models.py
│           ├── test_reset_step_state.py
│           ├── test_rewards.py
│           ├── test_anti_cheat.py
│           ├── test_seed_reproducibility.py
│           ├── test_invalid_actions.py
│           └── test_rollouts.py
├── training/
│   ├── train_grpo.py
│   ├── rollout.py
│   ├── reward_funcs.py
│   ├── eval_before_after.py
│   ├── trackio_utils.py
│   └── configs/
│       └── grpo_small.yaml
├── scripts/
│   ├── run_local.sh
│   ├── docker_build.sh
│   ├── docker_run.sh
│   ├── smoke_test.sh
│   ├── generate_scenarios.sh
│   └── push_space.sh
├── assets/
│   └── anti_overfitting_training_flow_diagram.png
└── outputs/
    ├── logs/
    ├── evals/
    └── rollouts/

If openenv init CyberSecurity_OWASP creates a different structure, preserve the generated structure and add the missing files around it.

---

Architecture overview

CyberSecurity_OWASP has 7 main components:

1. Policy Graph + Domain Sampler — samples users, roles, tenants, ownership, public routes, and business exceptions.
2. Template / Framework Randomizer — renders FastAPI-style apps with randomized layouts and naming.
3. A01 Bug Mutator — injects one authorization defect per scenario.
4. Fixture + Hidden Test Generator — creates users, resources, visible tests, and hidden invariant tests.
5. OpenEnv Server — exposes typed Action, Observation, and State through reset, step, and state.
6. LLM Agent + LoRA — one model performs discover + patch.
7. Deterministic Reward Engine — hidden tests score exploit blocking, normal-flow preservation, patch quality, and anti-cheat.

An optional LLM reviewer may score rationale quality and ASVS/OWASP mapping only. It must not provide the primary reward.

---

Scenario compiler requirements

Policy Graph + Domain Sampler

The policy graph is the source of truth. It must define:

• users;
• tenants;
• roles;
• resources;
• ownership relationships;
• role permissions;
• public routes;
• business exceptions.

Initial domains:

Domain	Example resources	Example policy rule
invoices	invoices, payments, accounts	owner or billing admin can read invoice
support	tickets, comments, customer records	assigned agent can update ticket
projects	projects, documents, milestones	project member can read project docs
marketplace	orders, returns, seller records	buyer owns own orders; seller owns own listings
HR	employee profiles, reviews, payroll records	HR admin can read employee records

Template / Framework Randomizer

First version: FastAPI only. Still randomize structure so the model cannot memorize one app.

Randomize:

• path naming;
• parameter names;
• helper names;
• folder layout;
• route/service/auth split;
• fixture names;
• error messages within valid policy bounds.

Examples:

/routes/invoices.py
/api/billing.py
/controllers/accounts.py
/services/access.py
/authz/guards.py

A01 Bug Mutator

Inject exactly one primary bug per scenario.

Initial bug families:

Bug family	Defect	Desired repair pattern
BOLA/IDOR	resource ID lookup lacks owner/tenant check	check server-side owner/tenant relation
BFLA	privileged route lacks role/function check	add reusable role or permission guard
tenant leak	request/header tenant ID is trusted	derive tenant from authenticated principal or server-side mapping
JWT claim trust	mutable claim is treated as authoritative	verify against server-side user/role record
public-route trap	route is intentionally public	do not over-secure public allowlisted route

Fixture + Hidden Test Generator

Visible tests should check that the app boots and normal happy paths work.

Hidden tests must check:

• exploit request is blocked;
• legitimate owner flow still works;
• legitimate admin/support flow still works;
• public routes remain public;
• cross-tenant access is denied;
• randomized IDs/names defeat hardcoded patches;
• hidden tests, fixtures, oracle, and reward files are not modified.

Scenario Cache + Seeded Reset

Training should use cached scenarios for speed.

Recommended first cache:

Split	Seeds	Purpose
train	500–1,000	RL rollouts
validation	100–200	checkpoint selection and curriculum signal
hidden_eval	100–200	final generalization proof

Hold-Out Generalization Splitter

Hold out at least 4 dimensions:

1. domains;
2. policy graph shapes;
3. code layouts;
4. bug-family/domain combinations.

Example: train on invoices/support/projects, evaluate on marketplace/HR.

---

OpenEnv model definitions

Implement these in envs/CyberSecurity_OWASP/models.py.

from dataclasses import dataclass, field
from typing import Any, Literal
from openenv.core.env_server import Action, Observation, State

CyberSecurityOWASPPhase = Literal["discover", "patch", "done"]
CyberSecurityOWASPSplit = Literal["train", "validation", "hidden_eval"]

@dataclass
class CyberSecurityOWASPAction(Action):
    tool_name: Literal[
        "inspect_policy_graph",
        "list_routes",
        "read_openapi",
        "read_file",
        "search_code",
        "send_local_request",
        "compare_identities",
        "submit_diagnosis",
        "patch_file",
        "run_visible_tests",
        "submit_fix",
        "noop",
    ]
    arguments: dict[str, Any] = field(default_factory=dict)

@dataclass
class CyberSecurityOWASPObservation(Observation):
    phase: CyberSecurityOWASPPhase
    message: str
    task_brief: str
    visible_policy_hint: dict[str, Any] = field(default_factory=dict)
    workspace_summary: dict[str, Any] = field(default_factory=dict)
    available_actions: list[str] = field(default_factory=list)
    last_tool_result: str = ""
    last_action_valid: bool = True
    last_action_error: str | None = None
    visible_test_result: str | None = None
    reward_breakdown: dict[str, float] = field(default_factory=dict)
    done_reason: str | None = None

@dataclass
class CyberSecurityOWASPState(State):
    episode_id: str = ""
    task_id: str = ""
    seed: int = 0
    split: CyberSecurityOWASPSplit = "train"
    difficulty: int = 0
    domain: str = ""
    bug_family: str = ""
    phase: CyberSecurityOWASPPhase = "discover"
    step_count: int = 0
    max_steps: int = 40
    done: bool = False
    success: bool = False
    failure_reason: str | None = None
    finding_submitted: bool = False
    patch_submitted: bool = False
    accumulated_reward: float = 0.0
    last_reward: float = 0.0
    action_history: list[dict[str, Any]] = field(default_factory=list)
    reward_history: list[dict[str, float]] = field(default_factory=list)
    visible_facts: dict[str, Any] = field(default_factory=dict)
    hidden_facts: dict[str, Any] = field(default_factory=dict)
    metrics: dict[str, Any] = field(default_factory=dict)
    anti_cheat_flags: list[str] = field(default_factory=list)

---

Action design and phase gating

Actions must be explicit, typed, serializable, and constrained. Invalid actions must not crash the server.

Phase-gated tools

Phase	Allowed tools
discover	inspect_policy_graph, list_routes, read_openapi, read_file, search_code, send_local_request, compare_identities, submit_diagnosis, noop
patch	read_file, search_code, patch_file, run_visible_tests, send_local_request, submit_fix, noop
done	no state-changing tools; return stable done observation

Tool contracts

inspect_policy_graph : Returns public policy hints. Must not reveal hidden bug labels or hidden tests.

list_routes : Returns route method/path summaries from the generated app.

read_openapi : Returns generated OpenAPI metadata.

read_file : Reads editable workspace files only. Must block hidden tests, reward files, oracle files, and host files.

search_code : Searches editable workspace files only.

send_local_request : Sends a request to the local generated app only. Must block external URLs and host network access.

compare_identities : Runs the same local request as two generated users and summarizes behavioral differences.

submit_diagnosis : Accepts structured evidence of the suspected authorization bug. Required before patch phase unless curriculum level explicitly allows blind patching.

patch_file : Applies a bounded unified diff to editable app files only.

run_visible_tests : Runs visible tests only. Must not run or reveal hidden tests.

submit_fix : Triggers hidden evaluation.

---

Observation rules

Observations should provide enough information to act but must not leak the answer.

Include:

• current phase;
• task brief;
• visible policy hints;
• workspace summary;
• available tools;
• previous tool output;
• visible test output;
• public reward breakdown after terminal evaluation.

Do not include:

• hidden bug family if not meant to be visible;
• hidden test contents;
• hidden oracle;
• exact exploit path labels;
• hidden seed split labels that allow memorization;
• reward implementation details that allow proxy hacking.

---

State rules

State is the source of truth for deterministic replay and debugging.

Required state properties:

• reset(seed) must create a fresh independent state;
• same seed + same action sequence should produce same result;
• each WebSocket session must be isolated;
• step_count increments once per processed action;
• terminal states return stable done observations;
• hidden facts never appear in observations;
• all actions and reward breakdowns are stored for debugging.

---

Environment API contract

Implement in envs/CyberSecurity_OWASP/server/environment.py.

from openenv.core.env_server import Environment
from ..models import CyberSecurityOWASPAction, CyberSecurityOWASPObservation, CyberSecurityOWASPState

class CyberSecurityOWASPEnvironment(Environment):
    def __init__(self):
        super().__init__()
        self._state = CyberSecurityOWASPState()

    def reset(self) -> CyberSecurityOWASPObservation:
        ...

    def step(self, action: CyberSecurityOWASPAction) -> CyberSecurityOWASPObservation:
        ...

    @property
    def state(self) -> CyberSecurityOWASPState:
        return self._state

step(action) must follow this order:

1. If done, return stable done observation.
2. Validate action and phase permissions.
3. Increment step count.
4. Execute the tool.
5. Update state/history.
6. Run verifier if submit_diagnosis, run_visible_tests, or submit_fix.
7. Compute reward components.
8. Check terminal conditions.
9. Return observation, reward, and done through OpenEnv step result handling.

---

FastAPI server and client

envs/CyberSecurity_OWASP/server/app.py must use the OpenEnv FastAPI helper.

from openenv.core.env_server import create_fastapi_app
from ..models import CyberSecurityOWASPAction, CyberSecurityOWASPObservation
from .environment import CyberSecurityOWASPEnvironment

env = CyberSecurityOWASPEnvironment()
app = create_fastapi_app(env, CyberSecurityOWASPAction, CyberSecurityOWASPObservation)

envs/CyberSecurity_OWASP/client.py must parse StepResult, observations, rewards, done flags, and state into typed classes.

---

Deterministic reward engine

The reward engine is the task specification. Build it before training.

Maximum reward: 15.0.

Component	Reward
valid finding with reproducible local evidence	+2.0
finding tied to violated policy rule	+1.0
hidden exploit blocked	+5.0
legitimate owner/admin/support flows preserved	+3.0
public routes preserved	+1.0
patch is localized, reusable, and policy-aligned	+2.0
visible tests pass and app boots	+1.0
exploit still works	-6.0
legitimate flow regression	-5.0
public route incorrectly locked	-3.0
hardcoded user/tenant/resource fix	-4.0
modified tests, fixtures, oracle, or hidden files	-8.0
app does not parse or boot	-6.0

The reward breakdown must use stable keys:

reward_breakdown = {
    "discovery": discovery_reward,
    "security": security_reward,
    "regression": regression_reward,
    "public_routes": public_route_reward,
    "patch_quality": patch_quality_reward,
    "visible_tests": visible_tests_reward,
    "safety": safety_reward,
    "anti_cheat": anti_cheat_reward,
    "total": total_reward,
}

Do not let explanation quality dominate the reward. If using an LLM reviewer, keep it separate from total or cap it at a small non-primary auxiliary signal.

---

Verifier requirements

Create envs/CyberSecurity_OWASP/validators.py and envs/CyberSecurity_OWASP/server/reward_engine.py.

Required verifier functions:

def verify_finding(state: CyberSecurityOWASPState, finding: dict) -> dict:
    ...


def run_hidden_security_tests(state: CyberSecurityOWASPState) -> dict:
    ...


def run_hidden_regression_tests(state: CyberSecurityOWASPState) -> dict:
    ...


def detect_cheating(state: CyberSecurityOWASPState, action: CyberSecurityOWASPAction) -> list[str]:
    ...


def compute_reward(state: CyberSecurityOWASPState, action: CyberSecurityOWASPAction, verifier_result: dict) -> dict[str, float]:
    ...

Verifier tests must prove:

• correct patch receives high reward;
• exploit-only finding without patch does not complete the episode;
• deny-all patch fails regression tests;
• hardcoded patch fails randomized hidden tests;
• modified hidden files produce anti-cheat penalty;
• visible-test-only patch does not guarantee high reward;
• repeated intermediate actions cannot inflate reward indefinitely.

---

Anti-overfitting requirements

CyberSecurity_OWASP must prevent overfitting to one app or scenario.

Use all of these defenses:

Risk	Required defense
memorizes one app	many domains and templates
memorizes route names	randomized path, resource, parameter, helper names
memorizes bug location	vary route/service/auth layer placement
learns deny-all patch	hidden positive-flow and public-route tests
learns hardcoded patch	randomized users, tenants, resource IDs, role names
overfits visible tests	hidden invariant tests and held-out eval
overfits one bug family	curriculum-sampled bug mix
overfits one code layout	hold out entire layouts and domains
optimizes explanation only	deterministic reward is primary

Acceptance target: at least 20% of domain/layout/bug combinations must be held out from training.

---

Safety and cybersecurity boundaries

This is a defensive AppSec training environment.

Allowed:

• local generated app probing;
• authorization reasoning;
• secure patching;
• visible and hidden test execution;
• policy-to-code mapping;
• defensive vulnerability validation in sandbox.

Forbidden:

• real-world exploitation;
• credential theft;
• persistence/evasion/malware behavior;
• scanning external targets;
• bypassing real services;
• writing exploit instructions for systems outside the local generated lab.

send_local_request must only target the generated local app.

---

Curriculum controller

RL needs partial successes. Implement at least 3 difficulty levels.

level_0: BOLA/IDOR, small app, direct route, obvious policy hint
level_1: BFLA or tenant bug, moderate app, realistic distractors
level_2: JWT trust or nested tenant/resource route, multiple files, false-positive traps
level_3: held-out domain/layout/bug combo, harder naming, fewer hints

Curriculum signal:

if exploit_block_rate < 60%:
    increase level_0 and level_1 tasks
elif regression_rate > 20%:
    increase positive-flow and public-route traps
elif public_route_false_positive_rate > 10%:
    increase intentionally public route examples
elif validation_reward plateaus:
    increase unseen layouts and nested resources
else:
    increase difficulty by 1

---

Training requirements

Create a runnable minimal training script using HF TRL or Unsloth.

Required files:

training/train_grpo.py
training/rollout.py
training/reward_funcs.py
training/eval_before_after.py
training/trackio_utils.py
training/configs/grpo_small.yaml

Recommended first model:

Qwen/Qwen3-1.7B

Acceptable alternatives:

Qwen2.5-Coder-1.5B-Instruct
Qwen2.5-Coder-3B-Instruct

Use LoRA / QLoRA. Do not full-finetune unless explicitly required.

---

Rollout function requirements

training/rollout.py must run a full OpenEnv episode.

def rollout_once(trainer, env, tokenizer, dataset_prompt: str, max_steps: int = 40) -> dict:
    result = env.reset()
    observation = result.observation

    prompt_ids = []
    completion_ids = []
    logprobs = []
    reward_trace = []
    action_trace = []
    observation_trace = []

    for _ in range(max_steps):
        if result.done:
            break

        prompt = build_cybersecurity_owasp_prompt(observation, action_trace, observation_trace)
        rollout_output = generate_rollout_completions(trainer, [prompt])[0]
        action = parse_action_json(rollout_output["text"])

        result = env.step(action)
        observation = result.observation

        prompt_ids.extend(rollout_output["prompt_ids"])
        completion_ids.extend(rollout_output["completion_ids"])
        logprobs.extend(rollout_output["logprobs"])
        reward_trace.append(float(result.reward or 0.0))
        action_trace.append(action)
        observation_trace.append(observation)

    final_breakdown = getattr(observation, "reward_breakdown", {}) or {}
    return {
        "prompt_ids": prompt_ids,
        "completion_ids": completion_ids,
        "logprobs": logprobs,
        "reward_total": float(final_breakdown.get("total", sum(reward_trace))),
        "reward_discovery": float(final_breakdown.get("discovery", 0.0)),
        "reward_security": float(final_breakdown.get("security", 0.0)),
        "reward_regression": float(final_breakdown.get("regression", 0.0)),
        "reward_patch_quality": float(final_breakdown.get("patch_quality", 0.0)),
        "reward_anti_cheat": float(final_breakdown.get("anti_cheat", 0.0)),
        "success": bool(getattr(env.state(), "success", False)),
        "episode_length": len(action_trace),
    }

The prompt must require the model to output exactly one JSON action at a time.

Example action format:

{"tool_name":"read_file","arguments":{"path":"app/routes/invoices.py"}}

---

Reward functions for TRL

training/reward_funcs.py must expose separate reward functions for GRPO/PPO logging.

def reward_total(completions, **kwargs):
    return [float(x) for x in kwargs.get("reward_total", [0.0] * len(completions))]


def reward_security(completions, **kwargs):
    return [float(x) for x in kwargs.get("reward_security", [0.0] * len(completions))]


def reward_regression(completions, **kwargs):
    return [float(x) for x in kwargs.get("reward_regression", [0.0] * len(completions))]


def reward_patch_quality(completions, **kwargs):
    return [float(x) for x in kwargs.get("reward_patch_quality", [0.0] * len(completions))]


def reward_anti_cheat(completions, **kwargs):
    return [float(x) for x in kwargs.get("reward_anti_cheat", [0.0] * len(completions))]

---

GRPO training config

Use Trackio in GRPOConfig.

import os
from trl import GRPOConfig

output_dir = os.getenv("OUTPUT_DIR", "CyberSecurity_OWASP-qwen3-1.7b-grpo")
trackio_space_id = os.getenv("TRACKIO_SPACE_ID", "Humanlearning/CyberSecurity_OWASP-trackio")

grpo_config = GRPOConfig(
    output_dir=output_dir,
    report_to="trackio",
    trackio_space_id=trackio_space_id,
    logging_steps=1,
    save_steps=25,
    learning_rate=5e-6,
    num_train_epochs=1,
    per_device_train_batch_size=1,
    gradient_accumulation_steps=32,
    num_generations=6,
    max_prompt_length=4096,
    max_completion_length=768,
    use_vllm=True,
    vllm_mode="colocate",
    vllm_gpu_memory_utilization=0.2,
    gradient_checkpointing=True,
    gradient_checkpointing_kwargs={"use_reentrant": False},
    push_to_hub=False,
)

Start with small debug runs before scaling.

---

Trackio logging requirements

Trackio is mandatory for training and evaluation visibility.

Canonical Trackio Space:

https://huggingface.co/spaces/Humanlearning/CyberSecurity_OWASP-trackio

Use TRACKIO_SPACE_ID=Humanlearning/CyberSecurity_OWASP-trackio for training, evaluation, and smoke runs. This is separate from the OpenEnv HF Space Humanlearning/CyberSecurity_OWASP; do not send Trackio runs to the environment Space.

Run naming convention:

CyberSecurity_OWASP-<model>-<algo>-level<difficulty>-<YYYYMMDD-HHMM>-<git_sha>

Log these training metrics:

train/reward_total_mean
train/reward_discovery_mean
train/reward_security_mean
train/reward_regression_mean
train/reward_public_routes_mean
train/reward_patch_quality_mean
train/reward_visible_tests_mean
train/reward_safety_mean
train/reward_anti_cheat_mean
train/success_rate
train/exploit_block_rate
train/regression_preservation_rate
train/public_route_preservation_rate
train/invalid_action_rate
train/timeout_rate
train/safety_violation_rate
train/reward_hacking_suspected_rate
train/episode_length_mean
train/episode_length_p95
train/rollouts_per_second
train/tokens_per_second
train/loss
train/learning_rate
train/kl
train/grad_norm

Log these evaluation metrics:

eval/baseline_success_rate
eval/trained_success_rate
eval/absolute_success_improvement
eval/baseline_mean_reward
eval/trained_mean_reward
eval/absolute_reward_improvement
eval/heldout_success_rate
eval/heldout_mean_reward
eval/exploit_block_rate
eval/regression_preservation_rate
eval/public_route_preservation_rate
eval/anti_cheat_pass_rate
eval/invalid_action_rate
eval/timeout_rate
eval/safety_violation_rate
eval/mean_episode_length

Log these environment metrics:

env/reset_latency_ms
env/step_latency_ms
env/verifier_latency_ms
env/reward_latency_ms
env/scenario_compile_latency_ms
env/error_rate
env/task_difficulty
env/task_seed

---

Rollout artifact requirements

Save sampled rollouts under outputs/rollouts/.

Each rollout JSON must include:

{
  "run_name": "...",
  "episode_id": "...",
  "task_id": "...",
  "seed": 123,
  "split": "validation",
  "difficulty": 1,
  "domain": "invoices",
  "bug_family": "bola_idor",
  "actions": [],
  "observations": [],
  "reward_breakdown_by_step": [],
  "final_reward_breakdown": {},
  "total_reward": 0.0,
  "success": false,
  "failure_reason": null,
  "safety_violations": [],
  "anti_cheat_flags": []
}

Minimum artifacts:

• 10 baseline rollouts;
• 10 mid-training rollouts;
• 10 trained rollouts;
• 10 held-out evaluation rollouts.

---

Evaluation requirements

Create training/eval_before_after.py.

It must evaluate:

Metric	Required
baseline success rate	yes
trained success rate	yes
absolute success improvement	yes
baseline mean reward	yes
trained mean reward	yes
absolute reward improvement	yes
held-out success rate	yes
exploit-block rate	yes
regression-preservation rate	yes
public-route preservation rate	yes
invalid action rate	yes
anti-cheat pass rate	yes

Save output:

outputs/evals/<run_name>_eval_summary.json

Minimum hackathon target:

>= 50 evaluation episodes
>= 3 independently logged reward components
>= 1 held-out split
>= 1 baseline-vs-trained comparison
>= 1 anti-cheat evaluation

Preferred demo target:

mean reward improvement >= 30%
hidden exploit-block pass rate >= 70%
regression-preservation pass rate >= 80%
public-route preservation pass rate >= 90%
anti-cheat pass rate >= 95%

---

Testing requirements

Before training, all tests must pass.

Required tests:

test_models.py
test_reset_step_state.py
test_rewards.py
test_anti_cheat.py
test_seed_reproducibility.py
test_invalid_actions.py
test_rollouts.py

Implement at least 3 scripted policies:

random_policy: explores action space; should usually fail but not crash
bad_policy: tries invalid/cheating actions; should be penalized
oracle_policy: uses internal test-only access to solve; should get high reward

The oracle policy is only for tests and must never be exposed to the model during training.

---

Deployment requirements

The environment must run in these modes:

1. local Python / Uvicorn;
2. Docker container;
3. Hugging Face Space;
4. OpenEnv client over WebSocket.

Required commands:

# initialize if not already scaffolded
openenv init CyberSecurity_OWASP

# local development
uv sync
uv run server
curl http://localhost:8000/health

# Docker
openenv build -t CyberSecurity_OWASP:latest
# or:
docker build -t CyberSecurity_OWASP:latest -f envs/CyberSecurity_OWASP/server/Dockerfile .
docker run -p 8000:8000 CyberSecurity_OWASP:latest

# HF Spaces
openenv push --repo-id <username>/CyberSecurity_OWASP

# client install from Space
pip install git+https://huggingface.co/spaces/<username>/CyberSecurity_OWASP

Use WebSocket mode for training rollouts. HTTP endpoints are acceptable for debugging only.

---

Scaling rules

Before scaling training, confirm:

1. one manual episode works;
2. scripted oracle can solve easy seeds;
3. random policy does not crash;
4. 10 validation rollouts complete;
5. reward distributions make sense;
6. Trackio receives metrics;
7. rollout artifacts are saved.

Then scale gradually:

1 episode -> 10 episodes -> 50 episodes -> 100+ rollouts -> training run

For high-volume rollouts, prefer local Docker or Uvicorn over remote HF Spaces because local WebSocket sessions reduce latency and avoid Space limits.

Parallel Modal training runs

Parallel Modal GRPO runs are allowed only when they do not overwrite each other's evidence, checkpoints, scenario assignments, or Hub outputs.

Before launching another run:

1. Check active Modal apps:

uv run --extra modal modal app list

2. If a CyberSecurity_OWASP app is active, inspect it before launching:

uv run --extra modal modal app logs <app-id>

3. Use Modal CLI-level detach and the launcher detach flag together, otherwise the spawned GPU function may stop when the local entrypoint exits:

uv run --extra modal modal run --detach scripts/modal_train_grpo.py \
  --max-steps 300 \
  --dataset-size 64 \
  --num-generations 8 \
  --max-completion-length 768 \
  --difficulty 0 \
  --trace-log-every 10 \
  --seed-start 10000 \
  --detach

When running jobs in parallel:

• Give every run a distinct --seed-start range, spaced by at least 10,000 seeds unless a smaller controlled comparison is intentional.
• Keep CYBERSECURITY_OWASP_SCENARIO_CACHE_MODE=require; do not compile scenarios in the training hot path.
• Do not run prepare-cache --cache-force while any training job is active. Scenario-cache writes can invalidate or race training resets.
• Leave --push-to-hub off for parallel experiments unless each run has a unique --output-repo-id.
• Keep run names unique. The launcher timestamp normally handles this; set an explicit RUN_NAME only when it is globally unique.
• Use different Trackio run names but the same Trackio Space so reward, throughput, GPU utilization, invalid-action rate, and success metrics remain comparable.
• Treat the shared Modal volumes as shared infrastructure: model cache and scenario cache should be read-only during parallel training; run/checkpoint outputs must live under each run's unique output directory.
• If the goal is a clean reward comparison, keep model, difficulty, dataset-size, num-generations, max-completion-length, and reward config fixed, changing only seed-start or the one hyperparameter being tested.

---

README requirements

The README must explain:

• what CyberSecurity_OWASP models;
• why authorization repair is useful for LLM RL;
• action space;
• observation space;
• state fields;
• scenario generation;
• reward components;
• hidden tests;
• anti-overfitting safeguards;
• anti-cheat safeguards;
• curriculum;
• local/Docker/HF Spaces commands;
• training with TRL/Unsloth;
• before/after evaluation.

Include a demo narrative:

1. Baseline model attempts a generated A01 authorization repair episode.
2. Verifier shows whether it discovered the bug and whether the patch regressed normal flows.
3. RL training improves reward and pass rates.
4. Trained model handles held-out domain/layout seeds.
5. Anti-cheat tests prove it is not using deny-all, hardcoding, or fixture tampering.

---

Implementation workflow for Codex

When implementing this repo, follow this exact order:

1. Inspect existing structure and tests.
2. Create/update 00_PROJECT_BRIEF.md and 01_ARCHITECTURE.md if missing.
3. Define CyberSecurityOWASPAction, CyberSecurityOWASPObservation, and CyberSecurityOWASPState.
4. Implement a dummy OpenEnv server and client.
5. Implement scenario compiler with 1 domain and 1 BOLA/IDOR mutator.
6. Implement editable workspace generation.
7. Implement local request tool.
8. Implement visible tests.
9. Implement hidden verifier and reward engine.
10. Add anti-cheat checks.
11. Add tests for normal, failing, and cheating rollouts.
12. Add oracle, random, and bad scripted policies.
13. Add scenario cache and seeded splits.
14. Add 3 domains and 3 bug families.
15. Add GRPO training script.
16. Add Trackio logging.
17. Add before/after evaluation script.
18. Add HF Spaces deployment config.
19. Run tests and smoke tests.
20. Produce demo artifacts and README results.

Do not jump to training code before environment and verifier are correct.

---

Definition of done

CyberSecurity_OWASP is done only when all are true:

• reset(), step(action), and state work;
• actions, observations, and state are typed dataclasses;
• the environment runs locally;
• the environment runs in Docker;
• the environment is deployable to HF Spaces;
• there are at least 5 meaningful reward components;
• reward components are logged separately;
• hidden tests exist;
• anti-cheat tests exist and pass;
• scenario cache has train/validation/hidden-eval splits;
• at least 3 bug families exist;
• at least 3 domains exist;
• at least 3 scripted policies exist;
• Trackio is configured for training and evaluation;
• before/after evaluation exists;
• held-out evaluation exists;
• at least 40 rollout artifacts are saved;
• README explains environment, reward, training, and demo story;
• demo shows baseline behavior, trained behavior, reward improvement, and safeguards.

---

Final PR checklist

Every PR summary must answer:

1. What real-world workflow does this implement?
2. What does the agent observe?
3. What actions can the agent take?
4. What hidden state exists and why is it hidden?
5. What terminates an episode?
6. What exact checks prove success?
7. What are the reward components and ranges?
8. How could the model hack the reward?
9. What anti-cheat checks prevent that?
10. What tests prove the reward cannot be trivially hacked?
11. What baseline success rate did we observe?
12. What trained success rate did we observe?
13. What held-out success rate did we observe?
14. What Trackio run contains the evidence?
15. Does behavior improve, or only the reward proxy?
16. Is the environment ready for HF Spaces deployment?

---

Source grounding and credibility

Source	Why used	Credibility
OWASP Top 10 A01 Broken Access Control	Authorization bug taxonomy and prevention framing	8.5/10
OWASP ASVS	Access-control verification grounding	9/10
NIST SP 800-218 SSDF	Secure software development lifecycle grounding	9.5/10
Smith et al., ESEC/FSE 2015, “Is the Cure Worse Than the Disease?”	Peer-reviewed basis for hidden tests and repair-overfitting risk	9/10
OpenEnv build/deploy/training docs	Typed model, server, client, deployment, and training mechanics	8/10
Meta OpenEnv Hackathon criteria	Judging alignment and minimum requirements	8/10

---

Non-negotiable rule

A reward that can be hacked is worse than no reward. Build the verifier, hidden tests, anti-cheat tests, and held-out evaluation before scaling training.