docs/datasets.md

Datasets

The data pipeline creates a compact but complete medication-safety training substrate.

Sources

• Local structured drug knowledge.
• Synthetic patients generated from simulator priors.
• Easy/medium/hard scenario files.
• Retrieval corpus and local evidence index.
• Optional Hugging Face instruction data (tatsu-lab/alpaca) for format warm start.
• Optional DDI API augmentation.
• Optional web fallback scraping through allowlisted public health domains.

Generated Artifacts

• data/processed/normalized_drugs.parquet
• data/processed/drug_classes.parquet
• data/processed/interactions.parquet
• data/processed/graph_edges.parquet
• data/processed/patients_synthetic.parquet
• data/processed/retrieval_corpus.jsonl
• data/scenarios/scenarios_easy.jsonl
• data/scenarios/scenarios_medium.jsonl
• data/scenarios/scenarios_hard.jsonl
• data/processed/training_corpus_sft.json(.jsonl)
• data/processed/training_corpus_grpo_prompts.jsonl
• data/processed/training_corpus_summary.json

Rebuild

.venv/bin/python scripts/build_synthetic_patients.py
.venv/bin/python scripts/ingest_open_drug_sources.py
.venv/bin/python scripts/build_drug_knowledge.py
.venv/bin/python scripts/build_retrieval_index.py
.venv/bin/python scripts/build_scenarios.py
.venv/bin/python scripts/bootstrap_data.py
.venv/bin/python scripts/build_training_corpus.py --profile small --with-local --with-synthetic --with-hf

Use --enable-ddi-api and --enable-web-fallback only when network access and provenance review are available.