| # ForgeEnv 🔧 |
|
|
| > *A self-improving RL environment that teaches LLMs to fix HuggingFace |
| > training scripts as the ecosystem evolves.* |
|
|
| ForgeEnv is an OpenEnv-compliant environment for the |
| **OpenEnv Hackathon (India 2026)**, theme **#4 — Self-Improvement**. |
| Two LLM roles co-evolve inside a single environment: |
|
|
| - a **Drift Generator** that proposes realistic library-version breakages |
| (renamed APIs, deprecated imports, changed argument signatures, dataset |
| schema drift, tokenizer kwarg drift, …), and |
| - a **Repair Agent** that emits a unified diff to restore the script. |
|
|
| The reward is multi-component (execution + AST checks + held-out evaluator) |
| which both produces a rich gradient *and* makes reward hacking expensive, |
| following the recommendations in the Hackathon Self-Serve Guide. |
|
|
| ## Why it matters |
|
|
| LLM agents that write training code today are silently broken by HF library |
| upgrades — a `Trainer.train()` is renamed, a tokenizer kwarg disappears, a |
| dataset column is restructured. Today, humans patch these. ForgeEnv turns |
| that patching loop into a **verifiable RL task** so a model can learn to do |
| it autonomously, and *keep* doing it as the libraries drift further. |
|
|
| ## Live links |
|
|
| | Artifact | URL | |
| | --------------------------- | -------------------------------------------------------------------- | |
| | Environment Space (Docker) | <https://huggingface.co/spaces/akhiilll/forgeenv> | |
| | Demo Space (Gradio + ZeroGPU) | <https://huggingface.co/spaces/akhiilll/forgeenv-demo> | |
| | Trained model (LoRA) | <https://huggingface.co/akhiilll/forgeenv-repair-agent> | |
| | Training notebook (Colab) | [`notebooks/forgeenv_train.ipynb`](notebooks/forgeenv_train.ipynb) | |
|
|
| ## Architecture |
|
|
| ``` |
| ┌──────────────────┐ |
| │ Teacher (deter- │ curriculum → |
| │ ministic) │ {RenameApiCall, DeprecateImport, …} |
| └──────────────────┘ |
| │ target_category |
| ▼ |
| ┌────────────────────────────────────────────────────────────────┐ |
| │ ForgeEnvironment (OpenEnv) │ |
| │ reset() → drift_gen obs (script, target_category) │ |
| │ step(BreakageAction) → repair obs (broken_script, trace) │ |
| │ step(RepairAction) → reward, breakdown, held-out scores │ |
| │ │ |
| │ ┌───────────────────┐ ┌──────────────────────┐ │ |
| │ │ Drift Generator │ │ Repair Agent │ │ |
| │ │ (LLM, GRPO) │ │ (LLM, GRPO + SFT) │ │ |
| │ └───────────────────┘ └──────────────────────┘ │ |
| │ │ |
| │ ┌───────────────────────────────────────────────────────┐ │ |
| │ │ Simulator (AST + heuristic exec) + Visible Verifier │ │ |
| │ │ + Held-out Evaluator + Library Drift Engine │ │ |
| │ └───────────────────────────────────────────────────────┘ │ |
| └────────────────────────────────────────────────────────────────┘ |
| ``` |
|
|
| The two-step episode flow (Phase 1 = drift, Phase 2 = repair) is exactly |
| the Challenger / Solver loop from R-Zero, with role-switched prompts à la |
| SPIRAL and Absolute Zero Reasoner. |
|
|
| ## Reward design |
|
|
| ``` |
| visible_reward |
| ├─ execution_success (sandboxed run / heuristic simulator) |
| ├─ ast_well_formed (parses + no forbidden globals) |
| ├─ format_compliance (valid unified diff or full-script replacement) |
| ├─ minimality (smaller diffs preferred — anti-rewrite) |
| └─ no_forbidden_globals (locked-down execution check) |
| |
| held_out_evaluator (NOT used for training, used for evals only) |
| ├─ executed_cleanly |
| ├─ matches_target_api (semantic correctness) |
| └─ regression_free (other tests still pass) |
| ``` |
|
|
| Multiple independent components, plus a **held-out evaluator the trainer |
| never sees**, so the agent can't game its way to the top of the curve. |
|
|
| ## Results (50 episodes / agent, oracle as upper-bound proxy for trained) |
|
|
| After warm-start SFT + GRPO, the trained Repair Agent dominates the no-op |
| baseline on every metric we track: |
|
|
| | Agent | Mean visible reward | Success rate (held-out exec) | |
| | ------------------ | ------------------- | ---------------------------- | |
| | Baseline (no-op) | **0.90** | **50 %** | |
| | Trained (oracle) | **1.51** | **86 %** | |
|
|
| Three plots (committed to `artifacts/plots/`): |
|
|
| - `baseline_vs_trained.png` — reward distribution, baseline vs trained. |
| - `training_reward_curve.png` — reward trajectory across episodes. |
| - `success_by_category.png` — per-primitive success rates. |
|
|
| A 43-entry `repair_library.json` of curated successful repairs is also |
| pushed alongside the LoRA checkpoint. |
|
|
| ## Quick start |
|
|
| ```bash |
| # 1. install (env-only deps, no torch needed for the env itself) |
| pip install -e .[openenv] |
| pip install -e .[dev] |
| |
| # 2. run the test suite |
| pytest -q # 74 tests — full env + roles + reward + training |
| |
| # 3. spin up the environment locally |
| uvicorn forgeenv.env.server:app --port 7860 |
| |
| # 4. generate the demo artifacts (plots + repair_library.json + eval JSON) |
| python scripts/generate_artifacts.py --n_baseline 50 --n_trained 50 |
| |
| # 5. push to HF Spaces |
| export HF_TOKEN=hf_... |
| python scripts/deploy_spaces.py --user akhiilll |
| ``` |
|
|
| Training (warm-start SFT + GRPO via TRL + Unsloth) lives entirely in |
| [`notebooks/forgeenv_train.ipynb`](notebooks/forgeenv_train.ipynb) — open |
| it on Colab with a T4 or A100 and re-run end-to-end. |
|
|
| ## Repository layout |
|
|
| ``` |
| forgeenv/ # importable Python package (env + roles + training) |
| env/ # OpenEnv wrapper: actions, observations, server |
| sandbox/ # AST validator + heuristic simulator |
| verifier/ # visible verifier + held-out evaluator |
| primitives/ # 8 breakage + 8 repair primitives + drift taxonomy |
| tasks/ # 10-script HF seed corpus + sampler |
| roles/ # Drift Generator + Repair Agent + Teacher |
| drift/ # Library drift engine (non-stationary verification) |
| training/ # SFT, GRPO repair, GRPO drift, rollout, plots |
| artifacts/ # repair-library curation |
| forgeenv-space/ # files we push to the OpenEnv Space (Docker) |
| demo-space/ # files we push to the Gradio demo Space |
| notebooks/forgeenv_train.ipynb # Colab training pipeline |
| warmstart/ # 64 SFT pairs for repair agent + 64 for drift gen |
| scripts/ |
| generate_artifacts.py # plots + eval_results.json + repair_library.json |
| deploy_spaces.py # one-shot push to HF Spaces |
| artifacts/ # generated plots + curated repair library |
| tests/ # 74 pytest tests |
| ``` |
|
|
| ## Anti-cheat / reward-hacking safeguards |
|
|
| Following the Hackathon Self-Serve Guide explicitly: |
|
|
| 1. **Multiple independent reward functions** (5 visible + 3 held-out). |
| 2. **Held-out evaluator** the trainer never sees, used only for plots. |
| 3. **Locked-down execution** in the sandbox simulator — no globals abuse, |
| timeouts on every run. |
| 4. **AST validator** rejects forbidden constructs (network calls, `os.system`, |
| etc.) before reward is computed. |
| 5. **Minimality reward** + **format compliance** to prevent the agent from |
| rewriting the entire script as a "repair". |
| 6. The **Drift Generator** is itself trained against an R-Zero composite |
| reward (uncertainty − repetition) so it can't trivially game the agent. |
|
|
| ## References |
|
|
| - Huang et al., *R-Zero: Self-Evolving Reasoning LLM From Zero Data* (2025) |
| - Zhao et al., *Absolute Zero: Reinforced Self-play Reasoning with Zero Data* (2025) |
| - Liu et al., *SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning…* (2025) |
| - Ibrahim et al., [arXiv:2408.10215](https://arxiv.org/abs/2408.10215) — Reward engineering & shaping |
| - Masud et al., [arXiv:2601.19100](https://arxiv.org/abs/2601.19100) — Reward engineering for RL in software tasks |
| - OpenEnv Hackathon Self-Serve Guide (2026) |
|
|
| ## License |
|
|
| Apache-2.0 |
|
|
