README.md

---
title: AgentDebugger Env 🐛
emoji: 📈
colorFrom: yellow
colorTo: green
sdk: docker
app_port: 8000
pinned: false
---

# AgentDebuggerEnv 🐛

> **Benchmarking Agentic Reasoning through the Iterative Hypothesis-Test-Fix Loop.**

An OpenEnv-compliant environment designed for the **Meta + PyTorch + HuggingFace OpenEnv Hackathon**. Unlike static code-repair benchmarks, **AgentDebuggerEnv** focuses on the *trajectory* of an agent's reasoning—measuring how effectively an agent forms hypotheses, observes failures, and iterates toward a solution in a live execution sandbox.

---

## 🚀 Overview

Debugging is one of the highest-leverage cognitive tasks in software engineering. Modern LLM agents often struggle with:
-   **Red Herrings**: Following misleading error messages to the wrong function.
-   **Stagnant Iteration**: Repeating the same failed fix attempt instead of updating their hypothesis based on new output.
-   **Concurrency Failures**: Failing to detect or fix non-deterministic bugs (race conditions).

**AgentDebuggerEnv** makes these failures measurable and scorable. The environment provides a live, sandboxed feedback loop where agents submit complete code fixes and receive real-time execution results.

---

## 🛠️ Core Mechanics: The Feedback Loop

The environment follows the standard OpenEnv interface (`reset`, `step`, `state`) but enforces a strict **Hypothesis-Test-Fix** cycle:

1.  **Hypothesis**: The agent must state its theory about the bug before every fix attempt. 
2.  **Execution**: The submitted code is executed in a secure sandbox with hard timeouts.
3.  **Observation**: The agent receives the actual `stdout` + `stderr` from the test suite, not just a binary pass/fail.
4.  **Reward**: Dense reward signal is provided at every step, scaling with test progress and hypothesis accuracy.

---

## 📁 Tasks & Difficulty

The environment includes three standardized tasks designed to test different facets of agentic reasoning:

| Task | Difficulty | Core Challenge |
| :--- | :--- | :--- |
| **Easy** | Easy | **Off-by-One**: Simple logic bug with an explicit, high-signal error message. |
| **Medium** | Medium | **Red Herring**: Interdependent functions where the error manifests far from the root cause. |
| **Hard** | Hard | **Race Condition**: A concurrency bug that is invisible to sequential tests. Agent must design a concurrent test to surface it. |

---

## ⚙️ How It Works (Spec Compliance)

### Data Models
- **Observation**: Includes `buggy_code`, `test_suite`, `previous_attempts` (full history), and `current_error_output`.
- **Action**: Supports `submit_fix` (requires `hypothesis`), `query_context` (for deeper code analysis), and `give_up`.
- **Reward**: A multi-component reward including `test_progress`, `hypothesis_match`, and `efficiency_bonus`.

### Infrastructure
-   **FastAPI**: Exposes standard endpoints on port 8000.
-   **Docker**: Fully containerized and ready for HuggingFace Spaces.
-   **Security**: Robust AST-based filtering to prevent malicious code escape.
-   **Baseline Script**: Includes a reference `inference.py` script that uses the OpenAI client for benchmark evaluation.

---

## 📦 Quick Start

### Installation
```bash
git clone https://huggingface.co/spaces/shashaank/agentdebugger-env
cd agentdebugger-env
pip install -r requirements.txt
```

### Running Locally
```bash
# Start the environment server
uvicorn env.server:app --host 0.0.0.0 --port 8000

# Run the baseline inference (requires API key)
export API_BASE_URL="https://api.openai.com/v1"
export MODEL_NAME="gpt-4o"
export HF_TOKEN="your_key_here"
python inference.py
```

---

## 📊 Benchmarking Results (GPT-4o Baseline)

| Task | Grader Score | Solved |
| :--- | :--- | :--- |
| Easy | 0.85 | Yes |
| Medium | 0.50 | Mixed |
| Hard | 0.18 | No |

---

## 📜 License
MIT License. Created by **shashaank** for the Meta / PyTorch / HuggingFace OpenEnv Hackathon.