README.md

---
title: AgentDebugger-Env 🐛
emoji: 📈
colorFrom: yellow
colorTo: green
sdk: docker
app_port: 8000
pinned: true
license: mit
---

# AgentDebuggerEnv 🐛

> **Benchmarking Agentic Reasoning through the Iterative Hypothesis-Test-Fix Loop.**

**AgentDebuggerEnv** is an OpenEnv-compliant benchmarking environment designed for the **Meta + PyTorch + HuggingFace OpenEnv Hackathon**. Unlike static code-repair benchmarks that only measure the final output, AgentDebuggerEnv evaluates the *cognitive trajectory* of an agent: how it forms hypotheses, interprets execution failures, and iterates toward a solution in a secure, live sandbox.

---

## 📊 Baseline Performance

Tested with **GPT-4o** using the standard `inference.py` script:

- **Easy (0.85)**: Solved in 1-2 attempts; clear signal from error output.
- **Medium (0.50)**: Solved in ~4 attempts; agents must resist a red-herring authentication error.
- **Hard (0.18)**: Rarely solved; agents must proactively design concurrent tests to surface the hidden race condition.
- **Mean Score: 0.51**

*Measurements taken over multiple runs to account for LLM variance. See `openenv.yaml` for full metadata.*

---

## 🚀 The Core Philosophy

Traditional benchmarks (like HumanEval or MBPP) are "one-shot": the model sees a prompt and writes code. Real-world engineering is **iterative**.

AgentDebuggerEnv forces agents to operate in a **live feedback loop**:
1.  **Observe**: Analyze existing buggy code and initial test failures.
2.  **Hypothesize**: Explicitly state a theory about the root cause (scored for accuracy).
3.  **Act**: Submit a surgical fix or query the environment for more context.
4.  **Verify**: Observe real-time `stdout/stderr` from a sandboxed test suite execution.

---

## 🛠️ Technical Architecture

### 1. Robust Security Sandbox
Every submission is executed in a multi-layered isolated environment:
*   **AST Filtering**: A deep Abstract Syntax Tree (AST) pass analyzes submitted code before execution, blocking dangerous imports (`os`, `sys`, `subprocess`, `socket`, etc.) and preventing the override of security-critical builtins.
*   **Process Isolation**: Executes in a separate subprocess with strict resource limits (CPU/Memory) enforced via container runtime and execution timeouts (15s). Any attempt to hang the environment results in immediate termination.
*   **Thread Safety**: A specialized "Concurrency Sandbox" allows multi-threaded tests (essential for the Hard Task) while maintaining strict host-level security boundaries.

### 2. High-Fidelity Feedback
Instead of binary `Pass/Fail` bits, the environment returns the **raw execution stream**. This allows agents to:
*   Read stack traces.
*   See partial progress (e.g., "6 passed, 2 failed").
*   Detect timeouts and resource exhaustion.

---

## 📁 Task Suite & Reasoning Challenges

| Task | Difficulty | Reasoning Challenge | Why it's hard |
| :--- | :--- | :--- | :--- |
| **Easy** | 🟢 Easy | **Off-by-One** | Requires basic logic verification. The error message is high-signal. |
| **Medium** | 🟡 Medium | **Red Herring** | The symptom (MD5 hashing error) manifests far from the root cause. Agent must trace data flow backward. |
| **Hard** | 🔴 Hard | **Race Condition** | **Invisible to sequential tests.** The agent must reason that passing tests do *not* mean the code is correct, and design a concurrent stress test. |

---

## 📊 Professional Grading Methodology

Our graders don't just check if the code works at the end. They score the **process**:

*   **Sequential Correctness (40%)**: Does the fix pass the original unit tests?
*   **Hidden Strength (30%)**: Does the fix survive a high-concurrency (1000-thread) stress test? (Hard task only).
*   **Hypothesis Accuracy (20%)**: Did the agent correctly identify the bug? (NLP-based keyword matching against ground truth).
*   **Efficiency Bonus (10%)**: Did the agent solve it within 5 attempts?

---

## ⚙️ Installation & Usage

### 📦 Local Setup
```bash
git clone https://huggingface.co/spaces/shashaank0707/AgentDebugger-env
cd AgentDebugger-env
pip install -e .
```

### 🚢 Running the Environment
```bash
# Start the FastAPI server
uvicorn env.server:app --host 0.0.0.0 --port 8000
```

### 🤖 Running an Agent (OpenEnv Baseline)
```bash
export API_BASE_URL="https://api.openai.com/v1"
export MODEL_NAME="gpt-4o"
export HF_TOKEN="your_openai_key"
export ENV_BASE_URL="http://localhost:8000"
python inference.py
```

---

### 🔐 Environment Variables

| Variable | Description | Standard Fallback |
| :--- | :--- | :--- |
| `API_BASE_URL` | LLM API endpoint | `https://api.openai.com/v1` |
| `MODEL_NAME` | Model to evaluate | `gpt-4o` |
| `HF_TOKEN` | Auth token (or OpenAI key) | — |
| `OPENAI_API_KEY` | Alternative auth token | — |
| `ENV_BASE_URL` | Address of the FastAPI server | `http://localhost:8000` |

---

## 🔗 OpenEnv API Compliance

AgentDebuggerEnv implements the full OpenEnv specification:

*   `POST /reset`: Initialize a task (`{"task_id": "medium"}`).
*   `POST /step`: Submit an `Action` (supports `submit_fix`, `query_context`, `give_up`).
*   `GET /state`: Retrieve full episode history and current environment state.
*   `GET /health`: Standard health check for automated uptime monitoring.

---

## 📜 Metadata & License
*   **License**: [MIT](LICENSE)
*   **Author**: Shashaank (GitHub: @shasshaank, HF: @shashaank0707)
*   **Hackathon**: Meta + PyTorch + HuggingFace OpenEnv 2024

---

### ✅ Submission Integrity
- **Commit SHA**: `159a5faf82fc1ab3709f9674becf9a3ec55cf562`
- **Last Verified Sync**: 2026-04-08
- **Platform Match**: GitHub and HF Space are identical at this HEAD.