implementation_plan.md

# AgentDebuggerEnv — Implementation Plan

An OpenEnv-compliant debugging environment where AI agents fix broken code through iterative hypothesis-test-fix cycles. Submission for the **Meta + PyTorch + HuggingFace OpenEnv Hackathon**.

## User Review Required

> [!IMPORTANT]
> This is a large project with **15+ files** to create. The entire codebase needs to be built from scratch (only the README exists currently). Please confirm you'd like me to proceed with the full implementation.

> [!WARNING]
> The README specifies `huggingface_space: shashaank/agentdebugger-env`. You'll need to create this HuggingFace Space and deploy the Docker container there for the hackathon submission. I'll build everything locally; deployment is a manual step.

## Proposed Changes

The implementation follows the exact order from the README's Section 14 checklist. Each step depends on the previous.

---

### Step 1: Sandbox (`env/sandbox.py`) — Build & Test First

This is the most security-critical component. Every code execution goes through here.

#### [NEW] [sandbox.py](file:///Users/shashaankjain/Desktop/meta_hackathon/env/sandbox.py)

- `execute_code(code, test_code, allow_threading=False) → (str, bool, int)`
- AST-based import detection (not string matching) to block dangerous imports
- `BLOCKED_IMPORTS` list: os, sys, subprocess, socket, importlib, shutil, pathlib, glob, pickle, shelve, dbm, sqlite3, ftplib, http, urllib, requests, httpx, asyncio, multiprocessing, threading (unless `allow_threading=True`), ctypes, cffi, resource, signal, mmap, gc
- Write code + test_code to a temp file, run in subprocess with `timeout=10`
- Capture merged stdout+stderr
- Clean up temp files in `finally` block

#### [NEW] [test_sandbox.py](file:///Users/shashaankjain/Desktop/meta_hackathon/tests/test_sandbox.py)

- 5 required tests: timeout, os blocked, sys blocked, clean code runs, syntax error returns output

---

### Step 2: Data Models

#### [NEW] [models.py](file:///Users/shashaankjain/Desktop/meta_hackathon/env/models.py)

- `FixAttempt`, `Observation`, `Action`, `Reward` — all Pydantic v2 BaseModel subclasses
- Exact field names and types from README Section 3

---

### Step 3: Task Definitions

#### [NEW] [task_easy.py](file:///Users/shashaankjain/Desktop/meta_hackathon/env/tasks/task_easy.py)

- Binary search with `<` instead of `<=` bug
- 8-test suite, 7 pass initially, 1 fails (last element)
- Ground truth: `hypothesis_keywords`: ["left <= right", "termination", "last element", "off by one", "<="]

#### [NEW] [task_medium.py](file:///Users/shashaankjain/Desktop/meta_hackathon/env/tasks/task_medium.py)

- `hash_password`, `validate_password`, `authenticate_user` — bug is in `hash_password`
- 10-test suite, 6 pass, 4 fail (edge cases with hash mismatch)
- Red herring: error points to `authenticate_user` but bug is in `hash_password`
- Hypothesis must mention "hash_password" AND at least 1 other keyword

#### [NEW] [task_hard.py](file:///Users/shashaankjain/Desktop/meta_hackathon/env/tasks/task_hard.py)

- `ConnectionCounter` with race condition in `increment()`/`decrement()`
- 8 sequential tests all pass on buggy code
- Bug only surfaces under concurrent access
- `allow_threading=True` for this task

#### [NEW] [registry.py](file:///Users/shashaankjain/Desktop/meta_hackathon/env/tasks/registry.py)

- Maps `"easy"` / `"medium"` / `"hard"` → task config dict (buggy_code, test_suite, description, ground_truth, max_attempts, max_steps)

#### [NEW] [`__init__.py` files](file:///Users/shashaankjain/Desktop/meta_hackathon/env/__init__.py)

- `env/__init__.py` and `env/tasks/__init__.py`

---

### Step 4: Graders

#### [NEW] [base_grader.py](file:///Users/shashaankjain/Desktop/meta_hackathon/env/graders/base_grader.py)

- Abstract base class with `score()` method

#### [NEW] [grader_easy.py](file:///Users/shashaankjain/Desktop/meta_hackathon/env/graders/grader_easy.py)

- Standard formula: 0.60 test_pass_ratio + 0.20 efficiency + 0.15 hypothesis + 0.05 early_solve

#### [NEW] [grader_medium.py](file:///Users/shashaankjain/Desktop/meta_hackathon/env/graders/grader_medium.py)

- Same formula but with red herring detection: hypothesis mentioning only "authenticate_user" scores 0.0

#### [NEW] [grader_hard.py](file:///Users/shashaankjain/Desktop/meta_hackathon/env/graders/grader_hard.py)

- Custom weights: 0.40 original tests + 0.30 concurrent stress test + 0.20 hypothesis + 0.10 efficiency
- Runs a 1000-thread concurrent stress test against submitted code

#### [NEW] [test_graders.py](file:///Users/shashaankjain/Desktop/meta_hackathon/tests/test_graders.py)

- Determinism tests (same input → same output)
- Range tests (output always in [0.0, 1.0])

---

### Step 5: Environment Core

#### [NEW] [environment.py](file:///Users/shashaankjain/Desktop/meta_hackathon/env/environment.py)

- `DebuggerEnvironment` class with `reset(task_id)`, `step(action)`, `state()` methods
- `reset()`: loads task, runs buggy code through sandbox to get initial error output
- `step()`: routes by `action_type` — submit_fix → sandbox, query_context → return info, give_up → run grader
- All action rules from Section 3.2 implemented exactly
- Step-level reward calculation per Section 6.1
- Episode-level grader invocation on `done=True`
- Never crashes — all errors returned in `info["error"]`

#### [NEW] [test_environment.py](file:///Users/shashaankjain/Desktop/meta_hackathon/tests/test_environment.py)

- Unit tests for reset/step/state

---

### Step 6: FastAPI Server

#### [NEW] [server.py](file:///Users/shashaankjain/Desktop/meta_hackathon/env/server.py)

- `POST /reset` — body: `{"task_id": "easy"}`, returns Observation JSON
- `POST /step` — body: Action JSON, returns `{"observation", "reward", "done", "info"}`
- `GET /state` — returns full state dict
- `GET /health` — returns `{"status": "ok", "environment": "agentdebugger-env", "version": "1.0.0"}` with HTTP 200

---

### Step 7: Inference Script

#### [NEW] [inference.py](file:///Users/shashaankjain/Desktop/meta_hackathon/inference.py)

- Exact code from README Section 8 — already fully specified
- Root directory placement (not in `env/`)
- Reads env vars: `API_BASE_URL`, `MODEL_NAME`, `HF_TOKEN`, `ENV_BASE_URL`
- Uses `openai` Python client
- Saves `baseline_results.json`

---

### Step 8: Configuration & Deployment

#### [NEW] [openenv.yaml](file:///Users/shashaankjain/Desktop/meta_hackathon/openenv.yaml)

- Exact content from README Section 9

#### [NEW] [Dockerfile](file:///Users/shashaankjain/Desktop/meta_hackathon/Dockerfile)

- Exact content from README Section 10

#### [NEW] [requirements.txt](file:///Users/shashaankjain/Desktop/meta_hackathon/requirements.txt)

- Exact content from README Section 11

---

## Open Questions

> [!IMPORTANT]
> **Task Medium — The Hash Bug:** The README describes a bytes/str conversion bug in `hash_password` where `str()` wrapping adds `"b'"` prefix. I need to carefully design the `user_db` and test setup so that 6 tests pass and exactly 4 fail. The README leaves the exact test suite design for medium to the implementer. I'll design it to match the described behavior. Any preferences?

> [!IMPORTANT]
> **Hard Task Test Count:** The README says `tests_total: 8` for hard in `openenv.yaml`, but the hard task has 8 sequential tests (all pass) and the agent needs to design a concurrent test. The grader independently runs its own 1000-thread stress test. I'll keep `tests_total: 8` as the initial suite and the grader adds its own concurrent verification separately. Correct?

## Verification Plan

### Automated Tests
1. `pytest tests/test_sandbox.py -v` — All 5 sandbox tests pass
2. `pytest tests/test_graders.py -v` — Determinism and range tests pass
3. `pytest tests/test_environment.py -v` — Reset/step/state tests pass
4. Start server with `uvicorn env.server:app --port 8000`, then:
   - `curl http://localhost:8000/health` → 200 with correct JSON
   - POST `/reset` for each task → valid Observation
   - POST `/step` with various actions → correct responses
5. Variance self-check:
   - Dummy agent (submits `pass`) → scores < 0.15
   - Perfect agent (ground truth fix + correct hypothesis) → scores > 0.85 on easy

### Manual Verification
- Docker build: `docker build -t agentdebugger-env .`
- Docker run and health check
- User deploys to HuggingFace Space and runs `openenv validate .`