---
title: Morpheus RAG
emoji: 🧠
colorFrom: green
colorTo: blue
sdk: docker
app_port: 7860
pinned: false
---

# Morpheus

> Ask anything. Your documents answer.

A multi-tenant Retrieval-Augmented Generation platform. Upload PDFs, ask questions in natural language, get accurate answers with source citations — streamed token by token, scoped strictly to your documents.

**What makes it non-trivial:**
- Each user sees only their own documents, enforced at the database level via Supabase RLS
- Retrieval combines BM25 keyword search + pgvector semantic search + Cohere neural reranking
- Answers stream live via SSE with a Groq → Gemini → OpenRouter provider fallback chain
- PDF ingestion runs in the background via Celery + Redis — no request timeouts
- An on-device sklearn intent classifier decides whether to ask for clarification before searching
- The document classifier learns with every ingestion — no manual category management needed
- Repeated questions are served from a semantic Redis cache with version-invalidation on corpus change

---

## Project Structure

```
morpheus/
├── backend/
│   ├── main.py                  FastAPI entry point, rate limiter, Celery lifespan
│   ├── api/
│   │   ├── auth.py              /auth/verify, /auth/admin, /auth/me
│   │   ├── corpus.py            /corpus/* — list, rename, delete, recategorise
│   │   ├── ingest.py            /ingest/upload + /ingest/status/{task_id}
│   │   ├── query.py             /query — SSE streaming RAG endpoint
│   │   ├── admin.py             /admin — warmup, master key
│   │   └── frontend_config.py   /config — serves Supabase keys to frontend
│   ├── core/
│   │   ├── pipeline.py          Ingestion, retrieval, generation, memory, tree search
│   │   ├── providers.py         Groq / Gemini / OpenRouter wrappers with fallback
│   │   ├── classifier.py        3-stage document classifier (centroid → ensemble → LLM)
│   │   ├── intent_classifier.py sklearn intent model — retrains online every 25 queries
│   │   ├── cache_manager.py     Semantic Redis cache with version invalidation
│   │   ├── auth_utils.py        JWT decode helpers + require_auth_token Depends()
│   │   ├── config.py            All constants, model lists, tuneable settings
│   │   └── tasks.py             Celery task definition for background ingestion
│   └── services/
│       └── auth.py              Daily rotating password logic
├── frontend/
│   ├── index.html
│   ├── css/                     Design tokens, layout, components, graph, views
│   └── js/
│       ├── config.js            CONFIG object — the only file you change to deploy
│       ├── api.js               All fetch() calls (never call fetch() elsewhere)
│       ├── state.js             Global STATE object — single source of truth
│       ├── chat.js              Streaming token renderer, chat UI
│       ├── corpus.js            Document list, upload, category management
│       ├── graph.js             D3 force-directed knowledge graph
│       ├── inspect.js           Node detail panel
│       ├── ui.js                Shared helpers (toast, switchView, etc.)
│       └── main.js              Boot sequence + auth gate
├── shared/
│   └── types.py                 Pydantic models shared by API and pipeline
├── supabase/
│   ├── schema_backup.sql        Full schema — run this to set up a new Supabase project
│   ├── migrations/              Incremental migration scripts
│   └── rls/
│       ├── multi_tenancy_rls.sql    Row-level security policies
│       └── rpc_security_rls_audit.sql
├── Dockerfile                   Single-container build (API + frontend + Celery)
├── render.yaml                  Deploy backend to Render
├── vercel.json                  Deploy frontend to Vercel
└── .env.example                 Copy to .env and fill in your keys
```

---

## Local Setup

### 1. Prerequisites

- Python 3.11
- Redis running locally (`redis-server` or via Docker: `docker run -p 6379:6379 redis`)
- A Supabase project with the schema applied (see step 3)

### 2. Clone and install

```bash
git clone https://github.com/your-username/morpheus.git
cd morpheus
python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate
pip install -r requirements.txt
```

### 3. Set up Supabase

1. Create a project at [supabase.com](https://supabase.com)
2. Enable the `vector`, `uuid-ossp`, and `pgcrypto` extensions first (Database → Extensions)
3. Go to the SQL editor and run `supabase/schema_backup.sql` in full
4. Create two Storage buckets:
   - `rag-images` as a public bucket
   - `rag-models` as a private bucket
5. Optional but recommended: run `supabase/rls/rpc_security_rls_audit.sql`

For an existing older database, apply the files in `supabase/migrations/` and
the scripts in `supabase/rls/` as incremental upgrade helpers. In particular,
run `supabase/migrations/0007_security_advisor_hardening.sql` and
`supabase/migrations/0008_enable_missing_rls.sql` if Supabase Security Advisor
flags mutable search paths, the old category materialized view, permissive
`evaluation_logs` policies, or missing RLS on tenant-scoped tables. For a
brand-new project, `schema_backup.sql` is already the full source of truth.

### 4. Configure environment

```bash
cp .env.example .env
```

Edit `.env` with your actual keys. All required variables:

| Variable | Where to get it |
|---|---|
| `SUPABASE_URL` | Supabase → Settings → API |
| `SUPABASE_ANON_KEY` | Supabase → Settings → API |
| `SUPABASE_SERVICE_KEY` | Supabase → Settings → API |
| `SUPABASE_JWT_SECRET` | Supabase → Settings → API → JWT Settings |
| `OPENROUTER_API_KEY` | [openrouter.ai/keys](https://openrouter.ai/keys) |
| `GROQ_API_KEY` | [console.groq.com](https://console.groq.com) |
| `GEMINI_API_KEY` | [aistudio.google.com](https://aistudio.google.com) |
| `COHERE_API_KEY` | [dashboard.cohere.com](https://dashboard.cohere.com) |
| `REDIS_URL` | `redis://localhost:6379/0` for local dev |
| `MASTER_ADMIN_KEY` | Any secret string you choose |

> All LLM providers have a generous free tier. This project is designed to run at zero API cost.

### 5. Run the backend

In one terminal:
```bash
uvicorn backend.main:app --reload --port 8000
```

In a second terminal (Celery worker for background ingestion):
```bash
python -m celery -A backend.core.tasks worker --pool=solo --loglevel=info
```

### 6. Open the frontend

Open `frontend/index.html` in your browser. `config.js` already points to `http://localhost:8000`.

Create a user account via Supabase Auth (Auth → Users → Invite), then sign in.

### 7. Optional: pre-build ML assets

```bash
# Downloads embedding model, trains intent classifier
Morpheus_BUILD_ASSETS_MODE=full python -m backend.core.build_ml_assets

# Downloads embedding model only (faster, recommended for first run)
Morpheus_BUILD_ASSETS_MODE=light python -m backend.core.build_ml_assets
```

### 8. Seed the document classifier (after your first ingestion)

```bash
curl -X POST http://localhost:8000/api/v1/admin/warmup \
     -H "X-Admin-Key: YOUR_MASTER_ADMIN_KEY"
```

---

## Docker

Build and run the full stack as a single container:

```bash
docker build -t morpheus-rag .
docker run -p 8000:7860 --env-file .env morpheus-rag
```

> The Dockerfile is configured for HuggingFace Spaces (port 7860, non-root user). For local use, the container maps 8000 → 7860.

---

## Cloud Deploy (Free Tier)

### Backend → Render

1. Push this repo to GitHub
2. [render.com](https://render.com) → New → Web Service → connect your repo
3. Render reads `render.yaml` automatically
4. In the Render dashboard → Environment tab, add all your `.env` values
5. Add a Redis service: Render → New → Redis (free tier) → copy the internal URL to `REDIS_URL`
6. Deploy — copy your Render URL (e.g. `https://morpheus-api.onrender.com`)

### Frontend → Vercel

1. Update `API_URL` in `frontend/js/config.js` to your Render URL
2. [vercel.com](https://vercel.com) → New Project → connect your repo
3. Vercel reads `vercel.json` automatically
4. Deploy — copy your Vercel URL (e.g. `https://morpheus.vercel.app`)

### After both are deployed

Update `ALLOWED_ORIGINS` in the Render environment to your Vercel URL, then redeploy the backend.

---

## API Reference

All routes are prefixed `/api/v1/`. Interactive docs available at `/docs` (disabled in production via `DOCS_ENABLED=false`).

| Method | Path | Auth | Description |
|--------|------|------|-------------|
| GET | `/health` | None | Liveness check |
| GET | `/health/details` | None | Liveness + intent classifier status |
| GET | `/api/status` | None | Service info |
| POST | `/auth/verify` | None | Verify daily guest password |
| POST | `/auth/admin` | None | Verify master key, get today's code |
| GET | `/auth/me` | JWT | Identity check |
| GET | `/corpus/files` | JWT | List all ingested documents |
| GET | `/corpus/categories` | JWT | List distinct categories |
| POST | `/corpus/rename` | JWT | Rename a document |
| POST | `/corpus/recategorise` | JWT | Move document to new category |
| DELETE | `/corpus/{file_hash}` | JWT | Delete document and all chunks |
| POST | `/ingest/upload` | JWT | Upload PDF, queue ingestion |
| GET | `/ingest/status/{task_id}` | None | Poll ingestion progress |
| POST | `/query` | JWT | SSE streaming RAG query |
| POST | `/admin/warmup` | Admin key | Seed classifier centroids |
| GET | `/config` | None | Frontend Supabase config |

All protected endpoints require the header `X-Auth-Token: <supabase_jwt>`.

---

## Scaling Notes

The architecture is designed for independent layer scaling:

- **Ingestion throughput**: Celery workers are already in place — add more worker processes or point them at a larger Redis instance.
- **Query concurrency**: Increase Gunicorn workers in `render.yaml` from `-w 1` to `-w 2` or more when moving off the free tier.
- **Embedding cost**: Swap `EMBEDDING_MODEL` in `config.py` — the `FallbackEmbeddings` wrapper handles the rest.
- **LLM cost**: All model lists live in `config.py`. Replace free OpenRouter models with paid ones for higher rate limits.
- **New document types**: The document classifier learns them automatically. Run warmup after your first example of each type.