# System Behavior: Dataset Curation

> Living document. Updated by `/archive-spec` when features are completed.
> Last archived: F004 on 2026-03-24

---

## ADDED

### Curation script produces enriched question dataset
<!-- since: F004 -->

Running `python scripts/curate_questions.py` produces two JSON files (`data/questions/questions_train.json` and `data/questions/questions_eval.json`) containing 100+ enriched questions across 10 Spider databases. Each question record includes `question_id`, `question_text`, `database_name`, `gold_sql`, `gold_answer`, `answer_type`, `difficulty`, `tables_involved`, and `split` fields.

### Curation script downloads Spider SQLite databases on demand
<!-- since: F004 -->

Running `python scripts/curate_questions.py` downloads Spider SQLite database files into `data/databases/{db_id}/{db_id}.sqlite` for each configured database. Existing files are skipped.

### Curation script accepts validate-only mode
<!-- since: F004 -->

Running `python scripts/curate_questions.py --validate` validates the existing dataset files without downloading or re-generating. It checks field completeness, gold SQL execution, answer correctness, split integrity, and difficulty distribution. Returns exit code 0 if valid, 1 if invalid.

### Dataset provides train/eval split
<!-- since: F004 -->

The dataset is split into `questions_train.json` (approximately 70%) and `questions_eval.json` (approximately 30%) with no overlapping question IDs between the two files.

### Dataset covers multiple domains and difficulty levels
<!-- since: F004 -->

Questions span 10 Spider databases from diverse domains (education, entertainment, geography, automotive, HR, etc.) with difficulty distribution targeting approximately 40% easy, 40% medium, 20% hard based on the number of tables involved in each query.