---
title: OpenSleuth Trainer
emoji: 🛰️
colorFrom: red
colorTo: yellow
sdk: docker
app_port: 7860
pinned: false
suggested_hardware: t4-small
---

# OpenSleuth — Trainer

GPU Space that fine-tunes a small Qwen2.5 model with TRL **GRPO** to do
in-context program synthesis against the live OpenSleuth env.

## Pipeline

1. Wait for the env Space to report healthy.
2. Build a dataset of synthesis prompts: each row pairs one black-box
   function with N pre-sampled `(input, output)` probes drawn from the env.
3. Load `Qwen/Qwen2.5-0.5B-Instruct` in 4-bit + LoRA via `bitsandbytes` and
   `peft`.
4. Train with `trl.GRPOTrainer`, generating `num_generations=4` candidate
   completions per prompt and rewarding each against the env's verifier.
5. Persist the LoRA adapter to `/data/opensleuth-grpo` and (if `HF_TOKEN` is
   set as a Space secret) push to `anugrah55/opensleuth-qwen2.5-0.5b-grpo`.

## Reward

* `env_verifier_reward = env.score_submission(...) / 100` — the headline
  shaped reward, ranging roughly `[-0.5, +1.5]`.
* `format_reward` — small bonus for emitting a fenced ```python``` block
  whose `def` matches the target function name; helps the model converge on
  parseable output early.

## Hardware

`t4-small` is sufficient for 0.5B + LoRA + bnb-4bit. `a10g-small` will train
faster if available.

## Required Space secrets

* `HF_TOKEN` — write token if you want the LoRA adapter pushed to the Hub at
  the end of training.

## Tuning knobs

All knobs are exposed as env vars (defaults shown):

| Env var | Default | Meaning |
|---------|---------|---------|
| `ENV_URL` | env Space URL | OpenSleuth env to target |
| `MODEL_NAME` | `Qwen/Qwen2.5-0.5B-Instruct` | Base policy |
| `N_PER_FUNCTION` | `16` | Prompts per black-box function |
| `N_PROBES` | `6` | Probes per prompt |
| `NUM_GENERATIONS` | `4` | GRPO group size |
| `LEARNING_RATE` | `1e-5` | |
| `NUM_TRAIN_EPOCHS` | `1` | |
| `PER_DEVICE_BATCH_SIZE` | `1` | |
| `GRAD_ACCUM` | `8` | |