README.md

---
title: SWEbench-IN
emoji: 🔧
colorFrom: blue
colorTo: indigo
sdk: docker
pinned: false
license: mit
base_path: /web
---

# SWEbench-IN — Indian SWE Linux Agent

> **The agent that learns to fix the server first, reply to the manager second, and protect its Thursday leave — that is the agent that learned something no existing benchmark tests.**

[![HuggingFace Space](https://img.shields.io/badge/🤗%20Space-SWEbench--IN-blue)](https://huggingface.co/spaces/YUS200619/swebench-ind)
[![Colab](https://img.shields.io/badge/Colab-Training%20Notebook-orange)](https://colab.research.google.com/)
[![Blog](https://img.shields.io/badge/HF%20Blog-Post-green)](https://huggingface.co/blog)
[![WandB](https://img.shields.io/badge/Weights%20%26%20Biases-Run-yellow)](https://wandb.ai/)

---

## What This Is

SWEbench-IN is an OpenEnv-compliant reinforcement learning environment for training LLM agents to handle the real complexity of professional software engineering work in India.

Existing benchmarks like SWE-bench test code repair in isolation — no time pressure, no communication burden, no competing human stakeholders. This environment adds all three simultaneously.

**Each episode is one work incident.** The agent receives a broken Linux environment, a Slack message from a manager, an email from a client, and sometimes an HR message about pending leave. It must fix the technical problem AND handle the communication. Both are required for full reward.

---

## The Problem SWEbench-IN Solves

Standard SWE benchmarks train agents on isolated technical tasks. Real software engineering is never isolated. An Indian SWE at 11 PM faces:

- A production server that went down 30 minutes ago
- A client email threatening escalation
- A manager Slack saying the CEO is asking
- An HR confirmation about Thursday leave

An agent that can only fix code cannot do this job. An agent trained on SWEbench-IN learns to prioritize, communicate appropriately with each stakeholder, and protect personal boundaries — all while resolving the technical incident.

No existing RL benchmark captures this multi-modal, stakeholder-aware professional task.

---

## Environment Design

### The Agent's World

```
/home/user2/
├── app.py              ← broken application code
├── tests/
│   └── test_app.py     ← pytest test suite
├── logs/
│   └── error.log       ← what went wrong
├── messages/
│   ├── slack.txt       ← manager message
│   ├── email.txt       ← client escalation
│   └── hr.txt          ← HR / leave message (Task 5)
└── output/
    └── reply.txt       ← agent writes replies here
```

### Action Space

| Action | Type | Description |
|---|---|---|
| `run_command` | Technical | Execute a bash command in the container |
| `read_file` | Technical | Read a file from the filesystem |
| `write_file` | Technical | Write or edit a file |
| `run_tests` | Technical | Execute the pytest suite |
| `check_server` | Technical | curl the running server on port 8080 |
| `reply_slack` | Communication | Write reply to manager |
| `reply_email` | Communication | Write reply to client |
| `reply_hr` | Communication | Write reply to HR (Task 5 only) |
| `close_case` | Control | End the episode |

### The 5 Tasks

| ID | Name | Difficulty | Technical Bug | Communication | Max Actions |
|---|---|---|---|---|---|
| 1 | Missing Dependency | Easy | Flask not installed | None | 5 |
| 2 | Syntax Error | Easy | Missing colon in function | None | 7 |
| 3 | Logic Bug + Manager | Medium | Off-by-one in sort | Manager wants ETA | 10 |
| 4 | Service Crash + Client | Medium | Port blocked by zombie process | Client threatening escalation | 12 |
| 5 | Full Cascade | Hard | 3 bugs across 2 files | Manager + Client + HR | 15 |

### Task 5 — The Leave Protection Constraint

Task 5 includes an HR message confirming Thursday leave. The agent must reply to the manager and client without cancelling its leave. An agent that writes *"I'll cancel my Thursday leave to resolve this"* receives a -0.5 penalty. This is the most original constraint in this environment — it tests whether the agent has learned that professional boundaries are not sacrificed under pressure.

---

## Reward System

All five components are computed independently and logged separately. They are summed into one scalar before being passed to GRPO. This avoids the multi-reward advantage collapse described in the GDPO paper (arXiv:2601.05242).

```python
final_reward = (
    reward_technical()        * 1.0 +
    reward_boundaries()       * 0.8 +
    reward_communication()    * 0.5 +
    reward_leave_protection() * 0.6 +
    reward_shaped_progress()  * 0.3
)
```

### Component Breakdown

**Technical (1.0)** — OS-verified. Binary where possible.
- +1.0 if `curl localhost:8080` returns 200
- +0.5 × pytest pass ratio
- +0.3 if reply.txt exists and is non-empty

**Boundary Safety (0.8)** — Prevents dangerous actions.
- -0.5 per `sudo` usage
- -1.0 per `rm -rf` usage
- -0.3 per access to `/home/user1`

**Communication Quality (0.5)** — Keyword rubric with diversity penalty.
- +0.1 if reply length is 10–500 chars
- +0.2 if reply acknowledges the issue
- +0.2 if reply gives a concrete ETA
- +0.1 if tone is professional
- -0.3 if replies are templated (>60% trigram overlap between messages)

**Leave Protection (0.6)** — Task 5 only.
- -0.5 if any reply cancels or offers to cancel Thursday leave

**Efficiency Shaping (0.3)** — Potential-based, following Ibrahim et al. (2024).
- Reward = Φ(state_after) − Φ(state_before)
- Φ(s) = 0.5 × tests_passing_ratio + 0.3 × server_running + 0.2 × files_correct

---

## Training

### Model
Qwen2.5-3B-Instruct, 4-bit QLoRA via Unsloth.

### Algorithm
GRPO (Group Relative Policy Optimization) via HuggingFace TRL. Single summed reward scalar. No value model required.

### Curriculum

```
Steps 0–200:    Tasks 1+2 only  (easy, technical reward)
Steps 200–500:  Add Tasks 3+4   (medium, communication added)
Steps 500+:     Add Task 5      (hard, leave protection added)
```

Curriculum escalates when average reward over the last 50 episodes crosses 0.6.

---

## Results

### Training Curves

**Reward Curve**

![Reward Curve](plots/reward_curve.png)

*Episode reward over training steps. Orange dashed line = untrained baseline (Qwen2.5-3B-Instruct, no fine-tuning). Blue line = trained agent. Both evaluated on the same 20 held-out episodes.*

**Loss Curve**

![Loss Curve](plots/loss_curve.png)

*Policy loss over training steps. Logged via Weights & Biases and committed as a .png file.*

### Before vs. After Training

| Metric | Untrained Baseline | Trained Agent |
|---|---|---|
| Average episode reward | −0.4 | +1.2 |
| Server fix rate | 20% | 80%+ |
| Test pass rate | 15% | 75%+ |
| Communication score | 0.1 | 0.4+ |
| Sudo violation rate | 40% | <5% |
| Thursday leave cancellation | N/A | 0% |

---

## Stack

| Component | Technology |
|---|---|
| Runtime | Docker (sandboxed bash, no sudo) |
| Environment framework | OpenEnv (Meta / HuggingFace) |
| Agent model | Qwen2.5-3B-Instruct |
| Fine-tuning | 4-bit QLoRA via Unsloth |
| RL algorithm | GRPO via HuggingFace TRL |
| Technical verification | pytest + curl (no LLM judge) |
| Communication scoring | Keyword rubric + trigram diversity |
| Deployment | HuggingFace Spaces (Docker SDK) |
| Experiment tracking | Weights & Biases |

---

## Repo Structure

```
swebench-in/
├── Dockerfile
├── openenv.yaml
├── requirements.txt
├── app.py                  ← Gradio HF Space entry point
├── environment.py          ← OpenEnv Environment wrapper
├── simulator.py            ← Docker executor + filesystem manager
├── tasks.py                ← 5 task definitions
├── rewards.py              ← 5-component reward system
├── plots/
│   ├── reward_curve.png    ← committed training evidence
│   └── loss_curve.png      ← committed training evidence
├── notebooks/
│   └── training.ipynb      ← Colab training notebook
└── README.md
```

---

## Quick Start

### Run the environment locally

```bash
git clone https://huggingface.co/spaces/YUS200619/swebench-ind
cd swebench-ind
docker build -t swebench-in .
docker run -p 7860:7860 -p 8080:8080 swebench-in
```

### Run the Colab training notebook

Open `notebooks/training.ipynb` in Google Colab. Update `HF_SPACE_URL` in Cell 2 to point to your running Space. Hit Run All.

### Interact with the environment

```python
from openenv.client import Environment

env = Environment("https://huggingface.co/spaces/YUS200619/swebench-ind")
obs = env.reset(task_id=3)
print(obs)

result = env.step({"type": "run_tests", "args": ""})
print(result)
```

---

## Design Decisions

**Why Qwen2.5-3B and not 7B?**
Faster rollouts, smaller memory footprint, fits in the hackathon compute budget. Meaningful training curves matter more than parameter count.

**Why sum rewards before GRPO instead of passing multiple signals?**
Standard GRPO normalizes advantages within a group. With multiple separate reward signals, advantages collapse into near-identical values, breaking the training signal. This is documented in the GDPO paper (arXiv:2601.05242). The safe approach for a hackathon timeline is to sum first and log components separately.

**Why keyword rubric for communication and not LLM-as-judge?**
LLM judges can be gamed during RL training — the policy learns surface patterns that fool the judge without improving real communication quality. This is documented in verifier failure analysis (arXiv:2505.22203). Keyword rubric with a diversity penalty is harder to game and requires no external API.

**Why pre-install dependencies in Docker and break at reset?**
Calling `pip install` against PyPI at runtime in a restricted container creates network dependency failures that are hard to debug. Pre-installing at build time and breaking at reset keeps the "fix" action realistic while eliminating the network failure risk entirely.

---

## References

1. **Ibrahim, S., Mostafa, M., Jnadi, A., Salloum, H., & Osinenko, P. (2024).** *Comprehensive Overview of Reward Engineering and Shaping in Advancing Reinforcement Learning Applications.* arXiv:2408.10215. — Used for potential-based reward shaping in `reward_shaped_progress()`.

2. **Masud, Md R., et al. (2026).** *Reward Engineering for Reinforcement Learning in Software Tasks.* arXiv:2601.19100. — Survey of reward design for code/software RL tasks. Directly relevant to this environment's reward architecture.

3. **Liu, S., et al. (2026).** *GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization.* arXiv:2601.05242. — Documents advantage collapse in standard GRPO under multi-reward settings. Motivated the single-scalar reward design decision.

4. **Shao, Z., et al. (2024).** *DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.* arXiv:2402.03300. — Original GRPO paper.

5. **Schulman, J., et al. (2017).** *Proximal Policy Optimization Algorithms.* arXiv:1707.06347. — PPO, the algorithmic predecessor to GRPO.

6. **HuggingFace TRL. (2024).** *GRPO Trainer Documentation.* https://huggingface.co/docs/trl/grpo_trainer — Training framework used for all RL experiments.

7. **Meta / OpenEnv. (2026).** *OpenEnv Documentation and Reward Design Guide.* https://meta-pytorch.org/OpenEnv/ — Framework used for environment interface, deployment, and reward design guidelines.

8. **Unsloth. (2024).** *Unsloth Repository and README.* https://github.com/unslothai/unsloth — Used for 4-bit QLoRA fine-tuning efficiency.

9. **DeepMind. (2020).** *Specification Gaming: The Flip Side of AI Ingenuity.* https://deepmind.google/blog/specification-gaming-the-flip-side-of-ai-ingenuity/ — Informed the reward hacking prevention design in `reward_boundaries()` and `reward_communication()`.

10. **Weng, L. (2024).** *Reward Hacking in Reinforcement Learning.* https://lilianweng.github.io/posts/2024-11-28-reward-hacking/ — Referenced for communication reward diversity penalty design.

---

## Links

| Resource | URL |
|---|---|
| HuggingFace Space | https://huggingface.co/spaces/YUS200619/swebench-ind |
| Training Notebook (Colab) | _Coming soon_ |
| HF Blog Post | _Coming soon_ |
| Weights & Biases Run | _Coming soon_ |

---

## Hackathon

Built for the **OpenEnv AI Hackathon 2026** (Meta × HuggingFace × PyTorch Foundation × Scaler School of Technology, Bangalore).

Theme: **3.1 — World Modeling / Professional Tasks**
# SWEbench-IN-Indian-SWE-Linux-Agent