scripts/hpc/HANDOFF.md

ImmunoOrg 2.0 — Supercomputer Run Handoff (4-stage pipeline)

Hi! Thanks for running this. The whole thing is two commands and the cluster does the rest unattended. Total wall-clock: ~3-4 hours for the full 4-stage pipeline on a single A100/H100, or ~1-1.5 hours if you skip SFT.

What the pipeline produces:

• A trained LoRA defender (Qwen2.5-7B by default, configurable up to 14B/32B)
• 6+ evidence PNG charts (loss curves, baseline-vs-trained comparisons)
• A reusable training dataset on the HF Hub
• All artifacts auto-pushed to my HF account

---

What you'll need

• HF write token (sender will give you one, will look like hf_xxx...).
• GPU: A100 / H100 / V100 (32GB+). If you have multiple, even better.
• SLURM (most US clusters). PBS/Torque also works — see "Non-SLURM" below.
• Internet on GPU node for model download. Most clusters allow this. If not, see "Air-gapped" below.

---

Steps (literal copy-paste)

# 1. Clone the repo (~3 sec)
git clone https://github.com/Charannoo/immunoorg.git
cd immunoorg

# 2. One-time env setup (~5-8 min, downloads PyTorch + Unsloth + TRL + flash-attn)
bash scripts/hpc/setup_env.sh

# 3. Export the HF token
export HF_TOKEN="hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

# 4. Submit the entire 4-stage pipeline (returns immediately with 5 job IDs)
bash scripts/hpc/run_all.sh

That's it. SLURM will run all 4 stages in dependency order (each stage waits for the previous via --dependency=afterok:). When stage 4 finishes, every artifact is on the HF Hub and the sender can pull it from there.

---

What the pipeline actually does

Stage	Job	Resources	Time	What it produces
0	datasets	CPU only, 32G RAM	~25 min	1700+ scenarios + 200 heuristic trajectories + SFT data + GRPO prompt set, pushed to <user>/immunoorg-grpo-dataset
1	SFT warm-start	1 GPU, 64G RAM	~25 min	LoRA adapter trained on heuristic trajectories so the model already speaks the env's JSON format before GRPO starts
2	GRPO training	1+ GPU, 96G RAM	~90-120 min	Final LoRA adapter, evidence_grpo_training.png (loss + per-reward curves)
3	evaluation	1 GPU, 64G RAM	~30 min	100 episodes per family × 3 policies (random/heuristic/trained), produces evidence_eval_per_family.png and evidence_eval_summary.png
4	push artifacts	CPU only	~10 min	Pushes adapter + 6+ PNGs + raw logs to <user>/immunoorg-grpo-defender model repo

You can watch live with:

squeue -u $USER                       # job states
tail -f logs/stage*-*.out             # live training log

---

Customising

Want to use multiple GPUs (recommended if you have them)?

bash scripts/hpc/run_all.sh --multigpu 4

Stage 2 (GRPO) will be data-parallel across 4 GPUs via accelerate launch. Roughly cuts stage 2 time from 90 min to 25 min.

Want a bigger model (14B / 32B)?

Override before submitting:

export IMMUNOORG_MODEL="Qwen/Qwen2.5-14B-Instruct"      # needs A100 80GB or 2x A100 40GB
# or
export IMMUNOORG_MODEL="Qwen/Qwen2.5-32B-Instruct"      # needs 2x A100 80GB or 4x A100 40GB

bash scripts/hpc/run_all.sh --multigpu 2

Skip SFT (saves ~30 min, slightly weaker results)

bash scripts/hpc/run_all.sh --skip-sft

Custom partition / queue names

If your partition isn't called gpu and cpu:

bash scripts/hpc/run_all.sh --partition gpu-a100 --partition-cpu compute

Or set env vars: IMMUNOORG_PARTITION=gpu-a100 IMMUNOORG_PARTITION_CPU=compute.

Push to a different HF account

export HF_PUSH_REPO="your-username/immunoorg-defender"
export HF_DATASET_REPO="your-username/immunoorg-dataset"
bash scripts/hpc/run_all.sh

Common partition names by cluster

Cluster	GPU partition	CPU partition
TACC (Frontera/Lonestar)	rtx, v100	normal, development
NCSA Delta	gpuA100x4, gpuA40x4	cpu
NERSC Perlmutter	gpu, gpu_a100	regular_milan_ss11
Most universities	gpu, a100, h100	cpu, compute, general

Run sinfo -o '%P %G %D' if you're not sure.

---

Troubleshooting

sbatch: error: Invalid partition specified

→ sinfo -o '%P %G' shows real partition names. Pass --partition <name>.

Out of memory on GPU

→ Smaller model: export IMMUNOORG_MODEL="Qwen/Qwen2.5-3B-Instruct". → Smaller batch: export IMMUNOORG_GRPO_BATCH_SIZE=2 IMMUNOORG_SFT_BATCH_SIZE=2.

"RuntimeError: bf16 requires Ampere or newer"

→ V100 (Volta) detected. The pipeline auto-falls back to fp16 — should just work. If it doesn't, edit scripts/hpc/pipeline/02_grpo_train.py and force bf16=False, fp16=True.

Stage 0 / 4 want a CPU partition but cluster only has GPU

→ Submit them to GPU too: bash scripts/hpc/run_all.sh --partition-cpu gpu.

Air-gapped GPU node (no internet)

1. On the login node:

   bash scripts/hpc/setup_env.sh
   source .venv-hpc/bin/activate
   python -c "from transformers import AutoModelForCausalLM, AutoTokenizer; \
              AutoModelForCausalLM.from_pretrained('Qwen/Qwen2.5-7B-Instruct'); \
              AutoTokenizer.from_pretrained('Qwen/Qwen2.5-7B-Instruct')"
   

2. The model is now in ~/.cache/huggingface/. SLURM jobs reuse the cache.

Non-SLURM cluster (PBS/Torque, LSF, single-node interactive)

You can still run each stage manually inside an interactive GPU shell:

# Get an interactive GPU shell first (cluster-specific, e.g. PBS):
qsub -I -l select=1:ngpus=1:walltime=04:00:00

# Then:
source .venv-hpc/bin/activate
export HF_TOKEN="..."
python scripts/hpc/pipeline/00_generate_datasets.py
python scripts/hpc/pipeline/01_sft_warmstart.py
python scripts/hpc/pipeline/02_grpo_train.py
python scripts/hpc/pipeline/03_evaluate.py
python scripts/hpc/pipeline/04_push_artifacts.py

---

When it's done

Just message back:

• The five SLURM job IDs (run_all.sh prints them)
• Confirmation that the model + dataset URLs above contain the artifacts

Sender pulls everything from those URLs and re-deploys the HF Space.

Thanks again — this run is the missing piece for the hackathon submission. 🙏