src/ingestion/README.md

# RAG PoC — Ingestion

This folder contains Day-3 ingestion pipeline components.

Files:
- load_docs.py        : Markdown loader -> returns cleaned text + metadata
- chunker.py          : Deterministic whitespace chunker (approx tokens->chars)
- test_ingestion.py   : End-to-end loader -> chunker smoke test
- embeddings.py       : Offline deterministic pseudo-embedding stub (provider="local")
- save_embeddings.py  : Persist chunk embeddings to data/embeddings.jsonl
- search_local.py     : Local cosine-similarity retrieval against embeddings.jsonl
- data/embeddings.jsonl : Generated embeddings (JSONL)

Quick run (from `RAG-document-assistant/ingestion`, with `aienv` active):

1. Activate venv:
   source ~/aienv/bin/activate

2. Load & summarize docs:
   python3 load_docs.py /full/path/to/your/markdown_folder

3. End-to-end ingestion test:
   python3 test_ingestion.py /full/path/to/your/markdown_folder

4. Generate & save embeddings:
   python3 save_embeddings.py /full/path/to/your/markdown_folder local 64

5. Search locally:
   python3 search_local.py data/embeddings.jsonl "your query" 3 64

Notes:
- Replace `/full/path/to/your/markdown_folder` with your real path (e.g. /home/vn6295337/RAG-document-assistant/sample_docs).
- This pipeline uses a local pseudo-embedding for offline testing. Replace provider branches when ready to use real APIs.