specs/F001-VERIFICATION_SPEC.md

Verification Specification

Feature: F001 Generated from: specs/F001-VERIFICATION_INPUT.json Generated: 2026-03-24

---

1. Unit Tests

1.1 SQLAction Type

Test	Description	Input	Expected	Category
test_sqlaction_valid_describe	Create action with DESCRIBE type	SQLAction(action_type="DESCRIBE", argument="employees")	Fields set correctly	happy
test_sqlaction_valid_sample	Create action with SAMPLE type	SQLAction(action_type="SAMPLE", argument="orders")	Fields set correctly	happy
test_sqlaction_valid_query	Create action with QUERY type	SQLAction(action_type="QUERY", argument="SELECT 1")	Fields set correctly	happy
test_sqlaction_valid_answer	Create action with ANSWER type	SQLAction(action_type="ANSWER", argument="42")	Fields set correctly	happy
test_sqlaction_empty_argument	Argument is empty string	SQLAction(action_type="QUERY", argument="")	Accepted at type level (validation at step)	edge
test_sqlaction_whitespace_argument	Argument is only whitespace	SQLAction(action_type="QUERY", argument=" ")	Accepted at type level	edge
test_sqlaction_serialization	Round-trip JSON serialization	Create, serialize, deserialize	Identical fields	happy

1.2 SQLObservation Type

Test	Description	Input	Expected	Category
test_obs_all_fields_present	All required fields populated	Full observation construction	All fields accessible with correct types	happy
test_obs_done_false_initial	Initial observation has done=False	After reset	done == False	happy
test_obs_reward_none_nonterminal	Non-terminal obs has reward=None	After non-ANSWER step	reward is None	happy
test_obs_reward_set_terminal	Terminal obs has numeric reward	After ANSWER step	reward in {0.0, 1.0}	happy
test_obs_step_count_type	step_count is int	Any observation	isinstance(step_count, int)	happy
test_obs_budget_remaining_type	budget_remaining is int	Any observation	isinstance(budget_remaining, int)	happy
test_obs_action_history_list	action_history is a list of strings	After several steps	isinstance(action_history, list) and all elements are str	happy
test_obs_error_empty_on_success	error is empty string on success	After successful action	error == ""	happy
test_obs_schema_info_nonempty	schema_info is non-empty after reset	After reset	len(schema_info) > 0	happy

1.3 QuestionRecord Type

Test	Description	Input	Expected	Category
test_qr_all_fields	All fields populated from valid JSON	Parsed question JSON	All 8 fields present and correct types	happy
test_qr_difficulty_values	difficulty in allowed set	Various records	difficulty in {"easy", "medium", "hard"}	happy
test_qr_answer_type_values	answer_type in allowed set	Various records	answer_type in {"integer", "float", "string", "list"}	happy
test_qr_tables_involved_nonempty	tables_involved has at least one entry	Valid question	len(tables_involved) >= 1	happy

1.4 EpisodeContext Type

Test	Description	Input	Expected	Category
test_ctx_initial_step_count	step_count starts at 0	After reset	step_count == 0	happy
test_ctx_initial_budget	budget starts at configured value (default 15)	After reset	budget == 15	happy
test_ctx_described_tables_empty	described_tables starts empty	After reset	len(described_tables) == 0	happy
test_ctx_action_log_empty	action_log starts empty	After reset	len(action_log) == 0	happy
test_ctx_done_false_initial	done starts False	After reset	done == False	happy
test_ctx_gold_answer_computed	gold_answer is set after reset	After reset	gold_answer is not None	happy

1.5 SQLEnvironment.init

Test	Description	Input	Expected	Category
test_init_valid	Init with valid paths loads questions	Valid questions_path, db_dir	len(questions) > 0	happy
test_init_missing_questions_file	questions_path does not exist	"/nonexistent/q.json"	FileNotFoundError	error
test_init_missing_db_dir	db_dir does not exist	"/nonexistent/dbs/"	FileNotFoundError or ValueError	error
test_init_invalid_json	questions file is not valid JSON	Path to file with "{bad"	ValueError	error
test_init_empty_questions	questions file is empty array []	Path to file with "[]"	ValueError (no questions)	error
test_init_custom_budget	Custom step_budget is stored	step_budget=10	Environment uses 10-step budget	happy
test_init_default_budget	Default step_budget is 15	Omit step_budget	Budget is 15	happy

1.6 SQLEnvironment.reset

Test	Description	Input	Expected	Category
test_reset_returns_observation	Returns SQLObservation	reset()	isinstance(result, SQLObservation)	happy
test_reset_obs_has_question	Observation contains question text	reset()	len(obs.question) > 0	happy
test_reset_obs_has_table_names	schema_info contains table names	reset()	schema_info mentions at least one table	happy
test_reset_obs_no_columns	schema_info does NOT reveal columns initially	reset()	No column names in schema_info	happy
test_reset_obs_done_false	done is False	reset()	obs.done == False	happy
test_reset_obs_reward_none	reward is None	reset()	obs.reward is None	happy
test_reset_obs_step_count_zero	step_count is 0	reset()	obs.step_count == 0	happy
test_reset_obs_budget_full	budget_remaining equals configured budget	reset()	obs.budget_remaining == 15	happy
test_reset_obs_empty_history	action_history is empty	reset()	obs.action_history == []	happy
test_reset_obs_empty_error	error is empty string	reset()	obs.error == ""	happy
test_reset_seed_determinism	Same seed yields same question	reset(seed=42) twice	Same question both times	happy
test_reset_different_seeds	Different seeds can yield different questions	reset(seed=1) vs reset(seed=999)	Not necessarily same question (probabilistic)	happy
test_reset_no_seed_random	Without seed, question is randomly selected	reset() multiple times	At least one different question (probabilistic)	happy
test_reset_episode_id_set	Custom episode_id is reflected	reset(episode_id="ep-123")	Context has episode_id="ep-123"	happy
test_reset_missing_db_file	Question references nonexistent DB	Question with bad db_name	FileNotFoundError	error
test_reset_clears_previous_episode	Calling reset mid-episode starts fresh	reset, step, reset	Second reset has step_count=0, empty history	happy

1.7 SQLEnvironment.step

Test	Description	Input	Expected	Category
test_step_returns_observation	Returns SQLObservation	Any valid action	isinstance(result, SQLObservation)	happy
test_step_never_raises	step never raises exceptions	Various invalid inputs	Returns obs with error field, no exception	happy
test_step_increments_step_count	step_count increases by 1	DESCRIBE action	obs.step_count == 1	happy
test_step_decrements_budget	budget_remaining decreases by 1	DESCRIBE/SAMPLE/QUERY	obs.budget_remaining == 14	happy
test_step_answer_no_budget_decrement	ANSWER does not decrement budget	ANSWER action	budget unchanged from before	happy
test_step_appends_action_history	action_history grows each step	Two DESCRIBE actions	len(obs.action_history) == 2	happy
test_step_invalid_action_type	Unknown action_type returns error	action_type="UNKNOWN"	obs.error contains "Unknown action type" and lists valid types	error
test_step_empty_argument	Empty argument returns error	argument=""	obs.error contains "cannot be empty"	error
test_step_whitespace_argument	Whitespace-only argument returns error	argument=" "	obs.error contains "cannot be empty"	error

1.8 SQLEnvironment._execute_sql

Test	Description	Input	Expected	Category
test_exec_valid_select	Executes a valid SELECT	"SELECT 1"	Returns list with results	happy
test_exec_non_select_rejected	Non-SELECT is rejected	"DROP TABLE x"	ValueError raised	error
test_exec_insert_rejected	INSERT is rejected	"INSERT INTO t VALUES(1)"	ValueError raised	error
test_exec_update_rejected	UPDATE is rejected	"UPDATE t SET x=1"	ValueError raised	error
test_exec_delete_rejected	DELETE is rejected	"DELETE FROM t"	ValueError raised	error
test_exec_create_rejected	CREATE is rejected	"CREATE TABLE t(x INT)"	ValueError raised	error
test_exec_semicolon_multi	Multiple statements rejected	"SELECT 1; DROP TABLE t"	Error or only first executed safely	error
test_exec_syntax_error	Malformed SQL	"SELCET * FORM t"	sqlite3.OperationalError	error
test_exec_timeout	Query exceeding timeout is interrupted	Long-running query (e.g., cartesian join)	Error raised within ~5 seconds	error
test_exec_result_truncation	Results truncated to 20 rows	Query returning 100 rows	len(result) <= 20	edge
test_exec_empty_result	Query returning no rows	"SELECT * FROM t WHERE 1=0"	Empty list []	edge
test_exec_read_only	Connection is truly read-only	Attempt write via raw SQL	Error raised	error
test_exec_case_insensitive_select	select (lowercase) is accepted	"select 1"	Returns results	edge
test_exec_select_with_leading_whitespace	Leading whitespace before SELECT	" SELECT 1"	Returns results	edge
test_exec_select_with_comment	SQL comment before SELECT	"-- comment\nSELECT 1"	Handled correctly (accepted or rejected consistently)	edge

1.9 SQLEnvironment._handle_describe

Test	Description	Input	Expected	Category
test_describe_valid_table	Known table returns schema info	Existing table name	Result contains column names, types, row count	happy
test_describe_unknown_table	Unknown table returns error with list	"nonexistent_table"	Error containing "not found" and listing available tables	error
test_describe_case_sensitivity	Table name matching case behavior	Mixed case table name	Consistent behavior (either matches or provides helpful error)	edge
test_describe_updates_described_set	Described table is tracked	DESCRIBE a table	Table appears in described_tables set	happy
test_describe_repeated_table	Describing same table twice works	DESCRIBE same table twice	No error, returns same info	edge

1.10 SQLEnvironment._handle_sample

Test	Description	Input	Expected	Category
test_sample_valid_table	Known table returns sample rows	Existing table name	Formatted rows returned	happy
test_sample_unknown_table	Unknown table returns error	"nonexistent_table"	Error with available table list	error
test_sample_default_limit	Default limit is 5 rows	Omit limit parameter	At most 5 rows returned	happy
test_sample_empty_table	Table with no rows	Empty table	Returns empty or "no rows" result	edge

1.11 SQLEnvironment._handle_query

Test	Description	Input	Expected	Category
test_query_valid_select	Valid SELECT returns formatted results	"SELECT * FROM t LIMIT 3"	Formatted result string	happy
test_query_non_select	Non-SELECT returns error	"DROP TABLE t"	Error about SELECT-only	error
test_query_syntax_error	Bad SQL returns error	"SELCET *"	Error with sqlite3 message	error
test_query_timeout	Slow query returns timeout error	Expensive query	Error mentioning "timed out" and "5.0 seconds"	error
test_query_truncation_indicator	>20 rows shows truncation notice	Query returning many rows	Result indicates truncation occurred	edge
test_query_exactly_20_rows	Exactly 20 rows shows no truncation	Query returning 20 rows	No truncation indicator	edge

1.12 SQLEnvironment._handle_answer

Test	Description	Input	Expected	Category
test_answer_correct	Correct answer yields reward 1.0	Gold answer value	(True, 1.0)	happy
test_answer_incorrect	Incorrect answer yields reward 0.0	Wrong value	(False, 0.0)	happy
test_answer_case_insensitive	Case-insensitive comparison	"PARIS" vs gold "paris"	(True, 1.0)	edge
test_answer_whitespace_stripped	Leading/trailing whitespace stripped	" 42 " vs gold "42"	(True, 1.0)	edge
test_answer_sets_done	Episode is marked done after answer	Any answer	done == True	happy
test_answer_empty_string	Empty answer handled	""	(False, 0.0)	edge

1.13 SQLEnvironment._load_questions

Test	Description	Input	Expected	Category
test_load_valid_json	Valid questions JSON parsed	Path to valid file	list[QuestionRecord] with correct count	happy
test_load_file_not_found	Missing file raises error	"/nonexistent.json"	FileNotFoundError	error
test_load_invalid_json	Malformed JSON raises error	Path to "{bad"	ValueError	error
test_load_missing_fields	Record missing required field	JSON with missing gold_sql	ValueError or KeyError	error

1.14 SQLEnvironment._open_db

Test	Description	Input	Expected	Category
test_open_valid_db	Opens existing database	Valid db_name	sqlite3.Connection returned	happy
test_open_missing_db	Missing database raises error	"nonexistent_db"	FileNotFoundError	error
test_open_read_only	Connection is read-only	Valid db_name	Write operations fail	happy

1.15 SQLEnvironment._build_observation

Test	Description	Input	Expected	Category
test_build_obs_reflects_context	Observation matches episode context	Various context states	Fields match context values	happy
test_build_obs_schema_progressive	schema_info includes described columns	After DESCRIBE action	schema_info contains column details for described tables	happy
test_build_obs_schema_initial	Initial schema_info has only table names	Before any DESCRIBE	Only table names, no columns	happy

Run: pytest tests/unit/test_sql_environment.py -v

---

2. Integration Tests

Flow: Full Episode Lifecycle (Reset-Explore-Answer)

Step	Action	Expected	Verification
1	reset(seed=42)	Observation with question, table names, step=0, budget=15	Assert all initial observation fields
2	step(DESCRIBE, "employees")	Column info in result, step=1, budget=14	Assert result contains column names, step incremented
3	step(SAMPLE, "employees")	Sample rows in result, step=2, budget=13	Assert formatted rows in result
4	step(QUERY, "SELECT COUNT(*) FROM employees")	Count result, step=3, budget=12	Assert numeric result in output
5	step(ANSWER, "<correct_value>")	done=True, reward=1.0	Assert terminal state with positive reward
6	Verify no more steps accepted	Post-episode step attempt handled gracefully	Assert error or episode-over signal

Flow: Budget Exhaustion

Step	Action	Expected	Verification
1	reset() with budget=3	Observation with budget=3	Assert budget_remaining==3
2	step(DESCRIBE, "t1")	budget=2	Assert budget_remaining==2
3	step(DESCRIBE, "t2")	budget=1	Assert budget_remaining==1
4	step(DESCRIBE, "t3")	budget=0, done=True, reward=0.0	Assert terminal state, zero reward

Flow: Error Recovery Within Episode

Step	Action	Expected	Verification
1	reset()	Normal initial observation	Assert done==False
2	step(QUERY, "DROP TABLE x")	Error in obs.error, done=False	Assert error message, episode continues
3	step(QUERY, "SELCET * FORM t")	Error in obs.error, done=False	Assert SQL error message, episode continues
4	step(DESCRIBE, "nonexistent")	Error in obs.error, done=False	Assert "not found" error, episode continues
5	step(QUERY, "SELECT 1")	Success result, no error	Assert error=="" and result has data

Flow: Progressive Schema Discovery

Step	Action	Expected	Verification
1	reset()	schema_info has table names only	Assert table names present, no column details
2	step(DESCRIBE, "table_a")	schema_info now includes table_a columns	Assert column names for table_a appear
3	step(DESCRIBE, "table_b")	schema_info includes both table_a and table_b columns	Assert both tables' columns present

Flow: Seed Determinism

Step	Action	Expected	Verification
1	reset(seed=42)	Observation A	Record question and db
2	reset(seed=42)	Observation B identical to A	Assert same question_text and schema_info

Run: pytest tests/integration/test_episode_lifecycle.py -v

---

3. API Tests

POST /reset

Test	Request	Status	Response	Category
test_reset_no_params	{}	200	SQLObservation with done=False, budget=15	happy
test_reset_with_seed	{"seed": 42}	200	Deterministic SQLObservation	happy
test_reset_with_episode_id	{"episode_id": "ep-1"}	200	SQLObservation (episode_id reflected in context)	happy
test_reset_null_seed	{"seed": null}	200	Random question selected	happy
test_reset_server_error_missing_db	(DB file removed)	500	Error response indicating database not found	error
test_reset_response_schema	{}	200	Response contains all SQLObservation fields	happy

POST /step

Test	Request	Status	Response	Category
test_step_describe	{"action_type": "DESCRIBE", "argument": "employees"}	200	SQLObservation with schema result	happy
test_step_sample	{"action_type": "SAMPLE", "argument": "employees"}	200	SQLObservation with sample rows	happy
test_step_query	{"action_type": "QUERY", "argument": "SELECT 1"}	200	SQLObservation with query result	happy
test_step_answer	{"action_type": "ANSWER", "argument": "42"}	200	SQLObservation with done=True	happy
test_step_missing_action_type	{"argument": "x"}	422	Validation error	error
test_step_missing_argument	{"action_type": "QUERY"}	422	Validation error	error
test_step_empty_body	{}	422	Validation error	error
test_step_invalid_action_type	{"action_type": "HACK", "argument": "x"}	200	SQLObservation with error field set	error
test_step_without_reset	Call step before reset	200 or 500	Graceful error (not a crash)	error

Run: pytest tests/api/test_endpoints.py -v

---

4. E2E Tests

Scenario: Agent Solves Easy Question

Setup: Environment initialized with Spider questions and databases. Select an easy question with a known answer. Actions:

1. POST /reset with seed that selects an easy question
2. POST /step with DESCRIBE on relevant table
3. POST /step with SAMPLE on relevant table
4. POST /step with QUERY using correct SQL
5. POST /step with ANSWER providing correct value Expected: reward=1.0, done=True, total steps <= 15

Scenario: Agent Exhausts Budget

Setup: Environment initialized with budget=3. Actions:

1. POST /reset
2. POST /step with DESCRIBE x3 Expected: After 3rd step: done=True, reward=0.0, budget_remaining=0

Scenario: Agent Submits Wrong Answer

Setup: Environment initialized normally. Actions:

1. POST /reset
2. POST /step with ANSWER providing deliberately wrong value Expected: done=True, reward=0.0

Scenario: Agent Recovers From Errors

Setup: Environment initialized normally. Actions:

1. POST /reset
2. POST /step with QUERY "DROP TABLE x" (rejected)
3. POST /step with DESCRIBE "nonexistent_table" (error)
4. POST /step with valid QUERY
5. POST /step with correct ANSWER Expected: Error steps counted against budget, valid steps succeed, episode completes with reward=1.0

Run: pytest tests/e2e/test_full_episodes.py -v

---

5. Edge Cases Checklist

• Null/None seed and episode_id in reset
• Empty string argument for all action types
• Whitespace-only argument for all action types
• Very long SQL query (10KB+)
• SQL with unicode characters in string literals
• SQL with special characters (backticks, double quotes, brackets)
• Table name with special characters or spaces
• ANSWER with numeric value as string vs integer
• ANSWER with leading zeros ("042" vs "42")
• ANSWER with trailing decimal ("42.0" vs "42")
• Budget of 1 (single step before exhaustion)
• Budget of 0 (should episode immediately end?)
• Calling step after episode is done
• Calling reset multiple times without stepping
• Concurrent episodes (if supported)
• Database with zero tables
• Database with very large tables (performance)
• Question whose gold_sql returns empty result
• SELECT with subqueries, CTEs, UNION
• SQL injection attempts via argument field ("; DROP TABLE--)
• Very long table name in DESCRIBE/SAMPLE

---

6. Evidence Requirements

Category	Evidence Type	Example
Unit tests	pytest output	45 passed
Integration	pytest output	5 passed
API tests	pytest output or httpx test client	9 passed
E2E	pytest output with real Spider DB	4 passed
SQL sandbox	Demonstrate write rejection	ValueError: Only SELECT queries are allowed
Timeout	Demonstrate query interruption	Query timed out after 5.0 seconds
Budget	Demonstrate forced termination	done=True, reward=0.0 at budget=0
Answer comparison	Demonstrate case-insensitive match	"PARIS" == "paris" -> reward=1.0