docs/blog-material.md

Blog Material — Raw Knowledge Dump

Reference file for writing the SQLEnv blog post. Contains observations, training data, failure modes, and narrative threads extracted from 9 training runs. The blog outline is at docs/blog-outline.md, the draft at docs/blog-post.md.

Training Run Summary

Run progression (what each run taught us)

1. Run 1: SFT works, GRPO plateaus — no penalty for post-episode waste
2. Run 2: Qwen3 tokenizer expands dict args to null params — root cause of first collapse
3. Run 3: Without KL penalty, GRPO drifts structural tokens (<tool_response> instead of <tool_call>)
4. Run 4: KL penalty + reference model = OOM on L4
5. Run 5: KL too conservative with single-turn SFT — model only calls describe, never queries
6. Run 6: Multi-turn SFT breakthrough — first successful training, reward -0.1→0.7
7. Run 7: Repeat penalty, stable training, multi-table weakness exposed
8. Run 8: Thinking mode helps error recovery, introduces <think>assistant degenerate loop, OOM crash
9. Run 9: v2 continued training confirms ceiling — more epochs don't help medium questions

Key numbers

Metric	Value
Model	Qwen3-0.6B (target <0.5B per VISION.md, using 0.6B)
Hardware	Colab L4 (24GB VRAM)
SFT data	120 multi-turn trajectories, assistant_only_loss
Phase 1	435 easy questions, beta=0.04, 512 tokens, ~3.5h
Phase 2	467 easy+medium, beta=0.0, 1024 tokens, ~3.8h
Total training	~7.3h on L4 for both phases
Phase 1 reward	-0.1 → 0.5-0.7 (clear upward trend)
Phase 2 reward	oscillates 0.01-1.15, mean ~0.5 (no improvement over Phase 1)
Parse rate	>98% (valid tool-call JSON)
Eval accuracy (GRPO v1, N=50)	28-30% across runs (vs 0% base model)
Eval accuracy (GRPO v2, N=50)	24-32% across runs (within noise of v1)
Eval accuracy (base, 3-shot)	0% (format compliance only 19-20%)

Concrete Episode Examples (for blog)

Success: Simple aggregation (3 tool calls)

Question: "What is the total bonus given in all evaluations?"

1. describe("evaluation") → Employee_ID, Year_awarded, Bonus columns
2. query("SELECT SUM(Bonus) FROM evaluation") → 19500.0
3. answer("19500.0") → correct Reward: 1.15

Success: Error recovery with JOIN

Question: "Which employee received the biggest bonus?"

1. describe("employee") → Employee_ID, Name, Age, City
2. query("SELECT Name FROM employee ORDER BY Salary DESC LIMIT 1") → Error: no such column: Salary
3. describe("evaluation") → Employee_ID, Year_awarded, Bonus
4. query("SELECT Name FROM evaluation JOIN employee ON evaluation.Employee_ID = employee.Employee_ID ORDER BY Bonus DESC LIMIT 1") → Louis Deacon
5. answer("Louis Deacon") → correct Reward: 1.13

Success: Subquery with comparison

Question: "How many cars have greater accelerate than the one with the most horsepower?"

1. describe("cars_data") → Id, MPG, Cylinders, Edispl, Horsepower, Weight, Accelerate, Year
2. query("SELECT COUNT(*) FROM cars_data WHERE Accelerate > (SELECT Accelerate FROM cars_data ORDER BY Horsepower DESC LIMIT 1)") → 39
3. answer("39") → correct Reward: 1.15

Failure: Column name hallucination

Question: "What are all the makers and models?"

1. describe("cars_data") → Id, MPG, Cylinders, Edispl, Horsepower, Weight, Accelerate, Year (no "model" column)
2. query("SELECT Id, model FROM cars_data") → Error: no such column: model
3. describe("model_list") → ModelId, Maker, Model
4. query("SELECT Maker, Model FROM model_list") → correct results (truncated to 20 rows)
5. answer("1 | amc\n2 | audi\n3 | bmw...") → incorrect (truncated, missing rows) Model found the right data but answer was truncated by the 20-row display limit.

Failure: FK chain reasoning

Question: "Which city has most number of arriving flights?"

1. describe("AIRPORTS") → City, AirportCode, AirportName, Country, CountryAbbrev
2. query("SELECT CITY, COUNT(*) FROM AIRPORTS GROUP BY CITY ORDER BY COUNT(*) DESC LIMIT 1") → Albany | 4
3. answer("Albany") → incorrect Model counted airports per city, not flights per city. Needed to JOIN flights→airports on DestAirport=AirportCode, then GROUP BY City. The FK chain reasoning is missing.

Failure: Invented table

Question: "What are all details of students who registered but did not attend any course?"

1. describe("Student_Course_Registrations") → student_id, course_id, registration_date
2. query("SELECT ... FROM Student_Course_Registrations WHERE course_id NOT INSELECT ...") → syntax error (missing space)
3. describe("Course_Attendance") → Error: table not found. Available: Student_Course_Attendance Model invented "Course_Attendance" instead of reading the available table list. Cascading errors consumed the full step budget.

Reward Architecture Details

Three-layer structure

L1 Operational (every step):
  +0.02  exec_ok
  +0.01  new_info (unique SQL hash)
  -0.03  repeat penalty
  -0.02  step cost

L2 Progress (QUERY only):
  Delta from previous binned progress × 0.15
  Binned to {0, 0.25, 0.5, 0.75, 1.0}

L3 Terminal (ANSWER only):
  +1.0 correct, 0.0 wrong

Per-step clip: [-0.10, 0.15]

Why potential-based shaping matters

• Ng et al. (1999): F(s,s') = Φ(s') - Φ(s) preserves optimal policy
• Our delta progress IS potential-based with γ=1
• Cumulative caps are NOT potential-based (depend on trajectory history)
• Without this guarantee, agents learn to farm exploration rewards

Anti-farming mechanisms

• Hard budget (15 steps)
• Step cost (-0.02)
• Repeat penalty (-0.03)
• Terminal dominance (1.0 vs ~0.3 max exploration)
• Per-step clip [-0.10, 0.15]
• Post-episode penalty (-0.3)

Eval Results (N=50, 2026-04-11)

Comparison table (for blog, N=50 with retry, 2026-04-11, Run B)

Method	Accuracy	Avg Reward	Avg Steps	Parse Rate	Parse Fails	Budget Exhaust
zero-shot	0.0%	0.007	12.4	23.6%	434	38
1-shot	2.0%	0.061	14.0	17.0%	537	46
3-shot	0.0%	0.057	14.8	19.0%	551	49
GRPO v1	30.0%	0.386	3.5	100.0%	0	0
GRPO v2	24.0%	0.321	3.6	95.1%	8	1

Previous run (Run A, same day, same seed)

Method	Accuracy	Avg Reward	Avg Steps	Parse Rate	Budget Exhaust
zero-shot	0.0%	0.016	10.8	28.1%	31/50
1-shot	0.0%	0.031	14.8	15.6%	49/50
3-shot	0.0%	0.041	13.8	20.3%	44/50
GRPO v1	28.0%	0.355	4.0	95.0%	2/50
GRPO v2	32.0%	0.400	3.7	87.1%	2/50

Run-to-run variation (important for blog)

v1 and v2 show similar accuracy with noise at N=50: v1 scored 28% then 30%, v2 scored 32% then 24%. The difference between checkpoints is within run-to-run variation (6-8pp swing). For the blog, report both as "28-32% accuracy" or "roughly 30%" rather than claiming one is better. The meaningful comparison is GRPO (~30%) vs base model (0-2%), not v1 vs v2.

The variation comes from: (1) temperature sampling during generation, (2) question selection randomness at N=50, (3) v2's "Task complete." abstention pattern — on borderline questions, whether v2 guesses or abstains varies by run, causing larger accuracy swings.

Note: parse failures no longer end episodes — model gets a no-op DESCRIBE and continues. This gives base models the same step budget as trained models, but they waste it on repeated parse failures (avg 11-15 steps vs GRPO's 3.5-4.0).

Key observations from N=50 eval (with retry, 2 runs)

1. ~30% accuracy for GRPO vs 0-2% for base model across all conditions. v1 and v2 are statistically indistinguishable (28-30% vs 24-32% across runs).
2. Run-to-run variation is ~6-8pp — v1 scored 28% then 30%, v2 scored 32% then 24%. At N=50, don't over-interpret small differences between checkpoints.
3. Base model parse failure loop — without episode termination on parse failure, base models burn their entire 15-step budget repeating the same non-tool-call output (e.g., "- Single value: []" 11 times). 46-49/50 1-shot episodes hit budget exhaustion.
4. GRPO solves format compliance — 95-100% parse rate (v1) vs 17-28% for base. The trained model almost always produces valid <tool_call> JSON.
5. GRPO failure mode is SQL quality, not format — episodes with correct tool-call format but wrong SQL/answer dominate GRPO failures.
6. Extra turns don't help base models — more steps just mean more repeated failures. The fundamental gap is format compliance, not exploration budget.
7. 1-shot occasionally gets lucky — scored 2% in Run B (1/50 correct), 0% in Run A. At N=50, a single lucky episode swings accuracy by 2pp.

v2 vs v1: similar accuracy, more parse failures — behavioral shift

Across two runs, v1 and v2 show overlapping accuracy ranges (28-30% vs 24-32%). The difference is within run-to-run variation at N=50. However, v2 consistently shows more parse failures (8-22 vs 0-8), revealing a behavioral shift from continued training:

• v1 guesses more: v1 almost always calls answer(), even when uncertain. It submits wrong answers confidently (0 parse failures in Run B, 100% parse rate).
• v2 gives up on hard questions: v2 produces "Task complete." output after multiple failed queries instead of calling answer(), producing parse failures. v2 learned that some questions are unsolvable.
• Neither is clearly better: v2's caution helps on some runs (32% in Run A) and hurts on others (24% in Run B). The abstention behavior adds variance. For the blog, present them as equivalent (~30%) with a qualitative note about the behavioral difference.

The v2 parse failure pattern (from raw output):

[OK] DESCRIBE: country
[OK] QUERY: SELECT Name FROM country WHERE Population < ...
[PARSE FAIL] raw: Task complete.    ← gives up, doesn't call answer()
[PARSE FAIL] raw: Task complete.    ← repeats until budget

Compare v1 on the same type of question:

[OK] DESCRIBE: country
[OK] QUERY: SELECT Name FROM country WHERE ...
[OK] ANSWER: European cities and their names are: 42    ← wrong, but at least calls answer()

This is a form of calibrated uncertainty — v2 is better at knowing what it doesn't know. The incorrect answer reward of 0.0 (see learning #19 in session log) creates an avoid-answering incentive that v2 has partially internalized. A more generous incorrect-answer reward (e.g., +0.1 for attempting an answer in correct format) might recover these episodes.

For the blog narrative

The story is clear: GRPO teaches format compliance (0% → 95-100% parse rate) and strategic tool use (describe→query→answer in 3-4 steps). Base models waste 15 steps repeating parse failures. The ~30% accuracy ceiling (consistent across checkpoints and runs) comes from the 0.6B model's SQL reasoning capacity, not from the environment or training pipeline. The environment scales; the model doesn't. Report v1 and v2 as "roughly 30%" — the variation between runs is larger than the difference between checkpoints.

Format Mismatch Discovery (F011)

The three differences between eval and training

1. role:tool vs role:user — Qwen3 renders role:"tool" as <|im_start|>user\n<tool_response>...</tool_response>, role:"user" as <|im_start|>user\nplain text. Same role token, different content structure.
2. Structured tool_calls vs raw text — Training uses {"role":"assistant", "tool_calls":[{"function":{"name":"describe","arguments":"{...}"}}]}, eval was using {"role":"assistant", "content":"<tool_call>...</tool_call>"}.
3. No separator vs \n\n — TRL appends reset() return directly to user message. Eval had question\n\ntable_hint.

Impact

Before fix: 0% accuracy across ALL conditions (zero-shot, 1-shot, 3-shot, GRPO checkpoint). After fix: 10% zero-shot, 30% 1-shot, 50% 3-shot on base model. GRPO checkpoint still 10%.

Lesson

Eval format matching is not a nice-to-have. It's a prerequisite for ANY measurement. We spent time debugging model quality when the problem was plumbing.

Multi-Turn SFT — Why It's Critical

Per-turn SFT (broken)

• 347 examples, each one assistant turn
• ~50% were describe calls
• Model learned: "when asked a question, call describe"
• With KL penalty, model stayed anchored to this single-action policy
• Result: reward=0.00, all rollouts identical, advantage=0

Multi-turn SFT (working)

• 120 examples, each a full describe→query→answer trajectory
• assistant_only_loss via Qwen3 template patch ({% generation %} tags)
• Model learned: the SEQUENCE describe→query→answer
• With KL penalty, model explores within the multi-turn strategy
• Result: reward climbs to 0.7 in Phase 1

Template patch detail

Qwen3's chat template lacks {% generation %} tags needed by TRL for assistant_only_loss. We patch the template before SFT, restore original before GRPO (TRL does exact-match checks on template string in add_response_schema() and get_training_chat_template()).

The 0.6B Capacity Ceiling

What works at 0.6B

• Single-table queries: COUNT, SUM, AVG, MIN, MAX, GROUP BY, HAVING, ORDER BY, LIMIT
• Simple JOINs between 2 tables when FK is obvious (evaluation.Employee_ID = employee.Employee_ID)
• WHERE with LIKE, IN, BETWEEN, NOT IN subqueries
• Answer formatting: comma lists, pipe-delimited rows, [] for empty
• Error recovery: describe after SQL error, retry with correct column names
• sample tool usage (learned in Run 6, inconsistent later)

What breaks at 0.6B

• FK chain reasoning: 3+ table joins (Documents→Templates→Ref_Template_Types)
• Column name fidelity: reads FullName from describe, writes full_name in SQL
• Ambiguous column resolution: joins with same column name in both tables
• Complex subqueries: INTERSECT, EXCEPT, correlated subqueries with HAVING
• "stadium without concert" pattern: NOT IN with JOIN to get names
• Aggregate + GROUP BY + HAVING chains on multi-table joins

The hallucination pattern

The model describes a table and sees the exact column names. Then it writes SQL using pretrained column names that don't match. This isn't a memory problem — the schema is in the context window. It's a weight problem — pretraining biases override in-context information at 0.6B scale.

Thinking Mode Observations (Run 8)

Benefits

• Reasons through SQL errors: "no such column: airport_code" → <think> block → tries AirportCode
• Empty <think></think> on easy questions — token-efficient, emergent behavior
• Multi-step join planning in think blocks

New failure mode

~23% of rollouts: <think>assistant<think>assistant... repeating until token limit. Model fails to close </think> tag. Burns entire token budget with garbage.

OOM risk

Thinking blocks consume more tokens → higher peak memory during generation. Phase 2 crashed at step 182/467 with max_new_tokens=1280. Fix: reduce to 1024, or reduce num_generations from 4 to 3.

Narrative Threads for Blog

"The environment is the product"

From VISION.md: "SQLEnv is a reinforcement learning environment — not a text-to-SQL model. The environment is the product." The trained agent demonstrates that the environment works, but the contribution is the action space, reward architecture, and episode structure.

"Small model showing improvement proves more than large model with marginal gains"

A 0.6B model going from 0% to 10% accuracy with clear strategic behavior (describe→query→answer, error recovery) proves the environment produces learning signal. A 70B model with marginal gains would prove nothing about the environment.

"Analysts don't write perfect queries from scratch"

The hook. Frame the problem as: text-to-SQL evaluates guessing, not investigating. SQLEnv evaluates the process.

"Dense rewards need theory"

Potential-based shaping isn't just good practice — it's the guarantee that the agent optimizes for the right objective. Without it, we saw agents farming exploration rewards.

"Multi-turn SFT teaches strategy, not actions"

The difference between per-turn and multi-turn SFT is the difference between teaching vocabulary and teaching conversation.

References for Blog

• Ng, Harada, Russell (1999). Policy invariance under reward transformations. ICML.
• DeepSeek-AI (2025). DeepSeek-R1.
• Shao et al. (2024). DeepSeek-Math: GRPO.
• Sullivan et al. (2025/2026). GRPO is Secretly a Process Reward Model. ICLR 2026.
• Yu et al. (2018). Spider dataset.
• Li et al. (2023). BIRD benchmark.
• TIPS (2026). Turn-Level Information-Potential Reward Shaping.
• ToolRL (2025). Reward is All Tool Learning Needs.
• StepTool (2024). Step-grained RL for Tool Learning.

Showcase Notebook Transcripts (for blog)

Random agent episode (seed=7) — comedic failure

Question: "Count the number of paragraphs."

  SAMPLE Paragraphs         → reward=0.015
  SAMPLE Documents           → reward=0.015
  DESCRIBE Documents         → reward=0.015
  SAMPLE Documents           → reward=0.015  (repeat)
  DESCRIBE Documents         → reward=0.015  (repeat)
  DESCRIBE Documents         → reward=0.015  (repeat)
  DESCRIBE Templates         → reward=0.015
  SAMPLE Documents           → reward=0.015  (repeat)
  DESCRIBE Documents         → reward=0.015  (repeat)
  QUERY SELECT * FROM "Templates" LIMIT 5  → reward=0.0625
  DESCRIBE Documents         → reward=0.015  (repeat)
  DESCRIBE Paragraphs        → reward=0.015
  QUERY SELECT * FROM "Paragraphs" LIMIT 5 → reward=0.025
  QUERY SELECT * FROM "Documents" LIMIT 5  → reward=0.025
  ANSWER 76 | 20 | Robbin CV | y | None   → reward=0.000 (incorrect)

Total reward: 0.278. Used all 15 steps. Described Documents 5 times. Answered with a random row from the wrong table. Never wrote SELECT COUNT(*).

Oracle agent episode (seed=0) — clean solve

Question: "List the id of students who registered some courses and the number of their registered courses?"

  Step 1: DESCRIBE student_course_registrations
    → student_id INTEGER, course_id INTEGER, registration_date DATETIME
    → reward: +0.015

  Step 2: DESCRIBE students
    → student_id INTEGER, student_details VARCHAR(255)
    → reward: +0.015

  Step 3: QUERY
    SELECT T1.student_id, count(*)
    FROM students AS T1
    JOIN student_course_registrations AS T2
      ON T1.student_id = T2.student_id
    GROUP BY T1.student_id
    → 111|1, 121|2, 131|1, 141|2, 151|1, 161|1, 171|1
    → reward: +0.150

  Step 4: ANSWER [[111,1],[121,2],[131,1],[141,2],[151,1],[161,1],[171,1]]
    → correct
    → reward: +1.000

Total reward: 1.180. 4 steps, efficient. Exploration (L1+L2): 0.180, Terminal (L3): 1.000.

Baseline comparison (50 episodes each)

Policy	Success Rate	Avg Reward	Avg Steps
Random	0.0%	0.247	15.0
Oracle	100.0%	1.168	3.5

The gap between 0.247 and 1.168 defines the learning space. A trained agent lands somewhere between.

Reward constants (from server/reward.py)

+0.02   successful execution (no errors)
+0.01   new information (unique query)
-0.02   step cost (every action)
-0.03   repeat penalty (duplicate SQL)
[-0.10, +0.15]  per-step clipping range
+1.0    correct answer (terminal)
+0.0    wrong answer (terminal)

Terminal dominance: max exploration over 15 steps is ~0.3 (15 * 0.02 best case), while a correct answer adds 1.0.

Competition Context

OpenEnv Challenge (our target)

• Sponsors: PyTorch/Meta, HuggingFace, Unsloth
• Prize: $10K HF credits
• Judging: primarily blog-based
• Criteria: Creative OpenEnv use, Technical excellence, Storytelling, Open source demo, Green Agent wrapper
• Green Agent wrapper is an explicit judging criterion in the OpenEnv Challenge.

Deliverables

1. Environment on HF Hub — live at https://huggingface.co/spaces/hjerpe/sql_env (pushed 2026-03-29; Docker image at registry.hf.space/hjerpe-sql_env:latest)
2. Training notebooks/scripts on GitHub — notebooks/train_grpo.ipynb, notebooks/compare_methods.ipynb, notebooks/showcase_sqlenv.ipynb
3. Blog on HuggingFace — docs/blog-post-v1.md (draft)

TRL integration status (already done — do not re-research)

training/trl_adapter.py::SQLEnvTRL is a TRL-native environment_factory class: reset() + named tool methods describe() / sample() / query() / answer() with docstrings TRL uses to build the tool schema. The notebook passes it directly: GRPOTrainer(..., environment_factory=SQLEnvTRL, reward_funcs=[sql_env_reward_func]). The adapter runs SQLEnvironment in-process (not a WebSocket client to the HF Space) — intentional, because training opens N parallel sessions and the Space defaults to 1.

Competitive landscape

• SQL Repair (WALKMAN303) — buggy SQL fix, simpler than our multi-turn exploration
• Calendar Gym (Turing) — featured on HF blog, real-world framing + failure analysis
• OpenSec — cybersecurity with arXiv paper, adversarial evidence injection
• Our position: no interactive SQL exploration environment exists. SQL Repair is single-turn fix-it; we're multi-turn strategy discovery.

What winning entries do

1. Stakes framing — "this matters in production"
2. Concrete failure analysis with numbers
3. Contrast (random vs trained vs oracle)
4. Real data, not toy puzzles
5. Non-obvious insights from training

Green Agent Evaluator

What it is

OpenEnv's standardized evaluation wrapper pattern. A Policy protocol with evaluate(env, policy, n_episodes, seed) that runs any policy through the environment and reports aggregate metrics. Listed as an explicit judging criterion in the OpenEnv Challenge.

Implementation

• evaluation/policies.py — Policy protocol, evaluate() harness, RandomPolicy, EpisodeResult, EvaluationResult
• evaluation/oracle_policy.py — OraclePolicy baseline (runs gold SQL)
• tests/test_evaluation.py — 17 tests, all passing (unit + integration)

How it works

from sql_env.evaluation import evaluate, RandomPolicy, OraclePolicy

# Run 50 episodes with random policy
result = evaluate(env, RandomPolicy(seed=0), n_episodes=50, seed=0)
print(f"Success: {result.success_rate:.1%}, Reward: {result.avg_reward:.3f}")

# Run with trained policy (any class with select_action method)
result = evaluate(env, trained_policy, n_episodes=50, seed=42)

Where it's used

• notebooks/showcase_sqlenv.ipynb — Random vs Oracle baseline comparison
• notebooks/compare_methods.ipynb — All 5 conditions (zero-shot, 1-shot, 3-shot, GRPO v1, v2) run through evaluate()

Key design choices

• Error isolation: one episode crashing doesn't kill the run — logged as EpisodeResult(error=str(exc))
• Deterministic seeding: seed + episode_index per episode for reproducibility
• Protocol-based: any class with select_action(observation) -> action works — no inheritance required
• Aggregate + per-episode: EvaluationResult has both summary metrics and full episodes list for drill-down

For the blog

The Green Agent evaluator is the backbone of all evaluation. Every result in the comparison table flows through evaluate(). The trained GRPO model is wrapped in LLMToolCallingPolicy (which implements the Policy protocol) and evaluated identically to the Random and Oracle baselines. This is the standardized, reproducible evaluation pattern the challenge asks for.

Files to Reference

File	Relevance
docs/blog-outline.md	Section structure template
docs/blog-post.md	Current draft
docs/design-docs/reward-shaping-research.md	Reward theory + references
docs/exploration/grpo-training-session-log.md	All 9 runs detailed
vision/VISION.md	Product vision, success metrics
training/trl_adapter.py	Environment adapter code
notebooks/compare_methods.ipynb	Eval notebook
notebooks/train_grpo.ipynb	Training notebook