testing.md

Testing Guide

This document lists the environment variables you may need, the available routes, which params are required, and how to test each route quickly.

1) Environment Variables

Server runtime variables

Use these when running the FastAPI app (local or container):

• HOST: default 0.0.0.0 in Docker, localhost in app main()
• PORT: default 8000
• WORKERS: default 1 (used by container command)
• MAX_CONCURRENT_ENVS: default 32

Minimal local run on Windows PowerShell:

$env:HOST = "127.0.0.1"
$env:PORT = "8000"
$env:MAX_CONCURRENT_ENVS = "32"
uvicorn server.app:app --host $env:HOST --port $env:PORT

Inference script variables

Required:

• API_BASE_URL
• MODEL_NAME
• HF_TOKEN or OPENAI_API_KEY

Optional:

• ENV_BASE_URL (if omitted, inference.py launches from Docker image)
• PYTHON_ENV_IMAGE (default python_env-env:latest)
• MAX_STEPS (default 3)
• MAX_TASKS (default 3)
• INFERENCE_REPORT_PATH (default inference_results.json)
• TEMPERATURE (default 0)
• MAX_TOKENS (default 900)

Example:

$env:API_BASE_URL = "https://api.openai.com/v1"
$env:MODEL_NAME = "gpt-4.1-mini"
$env:OPENAI_API_KEY = "<your-key>"
$env:ENV_BASE_URL = "http://127.0.0.1:8000"
python inference.py

2) Task IDs You Can Use

• py-review-easy
• py-review-medium
• py-review-hard

3) Route Testing (Params + Examples)

Base URL:

http://127.0.0.1:8000

OpenEnv routes

POST /reset

• Required params: none
• Body: none

Test:

Invoke-RestMethod -Method Post -Uri "http://127.0.0.1:8000/reset"

POST /step

• Required params: none in query/path
• Required body shape:
  • operation: one of submit_findings, request_hint, finalize
  • findings: array (can be empty)
• Optional body fields:
  • patched_code: string or null
  • note: string or null

Minimal body example:

{
  "operation": "request_hint",
  "findings": []
}

Test:

$body = @{
  operation = "request_hint"
  findings  = @()
} | ConvertTo-Json

Invoke-RestMethod -Method Post -Uri "http://127.0.0.1:8000/step" -ContentType "application/json" -Body $body

Example with finding:

$body = @{
  operation = "submit_findings"
  findings = @(
    @{
      title = "Avoid eval on untrusted input"
      line = 2
      category = "security"
      severity = "critical"
      rationale = "eval can execute attacker-controlled code"
      recommendation = "Use json.loads instead"
      rule_id = "avoid-eval"
    }
  )
  patched_code = $null
  note = "first pass"
} | ConvertTo-Json -Depth 6

Invoke-RestMethod -Method Post -Uri "http://127.0.0.1:8000/step" -ContentType "application/json" -Body $body

GET /state

• Required params: none

Test:

Invoke-RestMethod -Method Get -Uri "http://127.0.0.1:8000/state"

GET /schema

• Required params: none

Test:

Invoke-RestMethod -Method Get -Uri "http://127.0.0.1:8000/schema"

WS /ws

• Use a websocket client to connect.
• No route params required.

Custom REST routes

GET /health

• Required params: none

Invoke-RestMethod -Method Get -Uri "http://127.0.0.1:8000/health"

GET /tasks

• Required params: none

Invoke-RestMethod -Method Get -Uri "http://127.0.0.1:8000/tasks"

GET /tasks/{task_id}

• Required path param: task_id

Invoke-RestMethod -Method Get -Uri "http://127.0.0.1:8000/tasks/py-review-easy"

POST /tasks/{task_id}/grade

• Required path param: task_id
• Body uses PythonReviewAction shape
  • operation defaults to submit_findings if omitted
  • findings array accepted
  • patched_code optional
  • note optional

$body = @{
  findings = @(
    @{
      title = "Avoid eval on untrusted input"
      line = 2
      category = "security"
      severity = "critical"
      rationale = "eval executes arbitrary code"
      recommendation = "Use json.loads"
    }
  )
} | ConvertTo-Json -Depth 6

Invoke-RestMethod -Method Post -Uri "http://127.0.0.1:8000/tasks/py-review-easy/grade" -ContentType "application/json" -Body $body

POST /review

• Required body field:
  • code: string
• Optional body field:
  • context: string

$body = @{
  code = "def f(x):`n    return eval(x)`n"
} | ConvertTo-Json

Invoke-RestMethod -Method Post -Uri "http://127.0.0.1:8000/review" -ContentType "application/json" -Body $body

GET /history

• Required params: none

Invoke-RestMethod -Method Get -Uri "http://127.0.0.1:8000/history"

DELETE /history

• Required params: none

Invoke-RestMethod -Method Delete -Uri "http://127.0.0.1:8000/history"

GET /config

• Required params: none

Invoke-RestMethod -Method Get -Uri "http://127.0.0.1:8000/config"

PUT /config

• Required params: none
• Body: PythonEnvConfig object
• All fields have defaults, so {} is valid for a reset-like update

Minimal test:

Invoke-RestMethod -Method Put -Uri "http://127.0.0.1:8000/config" -ContentType "application/json" -Body "{}"

Full body example:

$body = @{
  task_order = @("py-review-easy", "py-review-medium", "py-review-hard")
  max_steps_per_task = 4
  hint_penalty = 0.05
  false_positive_penalty = 0.08
  duplicate_penalty = 0.03
  patch_bonus_multiplier = 0.2
  max_history_entries = 50
} | ConvertTo-Json

Invoke-RestMethod -Method Put -Uri "http://127.0.0.1:8000/config" -ContentType "application/json" -Body $body

4) Quick Validation Commands

Run automated tests:

pytest -q

Run only API tests:

pytest -q tests/test_api.py