Add HF Jobs research — snapshot_download for private repos, script submission pattern

AGENTS.md

@@ -154,6 +154,57 @@ env = ActionMasker(Monitor(base_env), ...)  # DON'T DO THIS
 
 ---
 
+## How to Submit HF Jobs Correctly (Research Results)
+
+### Based on `huggingface.co/docs/hub/jobs-quickstart`:
+
+**DO NOT use `git clone` for private repos.**
+
+```python
+# WRONG ❌
+import subprocess
+subprocess.run(["git", "clone", "https://huggingface.co/spaces/e-rong/til-26-ae"])
+# Fails: git does not read HF_TOKEN env var
+
+# CORRECT ✅
+from huggingface_hub import snapshot_download
+snapshot_download(
+    repo_id="e-rong/til-26-ae",
+    repo_type="space",
+    local_dir="/app/til-26-ae-repo"
+)
+# snapshot_download auto-uses HF_TOKEN from environment
+```
+
+### Script Submission Pattern
+
+```python
+# Step 1: Write script to sandbox file first
+write(path="/app/train.py", content="...")
+
+# Step 2: Submit as file path (not inline)
+hf_jobs(
+    operation="run",
+    script="/app/train.py",  # ← sandbox file path, gets uploaded
+    dependencies=["torch", "sb3-contrib", "gymnasium", "pettingzoo",
+                  "numpy", "huggingface_hub", "pygame", "omegaconf",
+                  "mazelib", "imageio", "imageio-ffmpeg", "supersuit", "psutil"],
+    hardware_flavor="a10g-small",
+    timeout="6h",
+    namespace="E-Rong"  # ← bills to org
+)
+```
+
+The `script` parameter is a **sandbox file path** that gets uploaded to the job container. `dependencies` maps to `--with` in the `uv run` CLI.
+
+### Job Persistence
+- Jobs run on HF infrastructure, not in your sandbox
+- The sandbox can die — the job keeps running
+- Check logs with `hf_jobs(operation="logs", job_id="...")`
+- Job storage is ephemeral — **push checkpoints to Hub** (not just local)
+
+---
+
 ## Repo File Guide
 
 | File | What It Is |
@@ -166,6 +217,7 @@ env = ActionMasker(Monitor(base_env), ...)  # DON'T DO THIS
 | `phase2_final.zip` | Phase 2 complete model (when done) |
 | `ae_manager.py` | Inference code for the evaluation server |
 | `phase2_job.py` | Latest HF Job script (may need fixes) |
+| `smoke_test.py` | 5-minute validation job — test before any real job |
 | `train_all_phases.py` | Original training script |
 
 ---