| # CORTEX — Agentic Graph RAG Platform |
|
|
| > **CORTEX** is a production-grade, agentic Knowledge Graph platform that transforms unstructured documents and web content into an intelligent, queryable knowledge graph — with a full-featured React UI, streaming AI chat, real-time graph visualization, simulation personas, and deep ontology governance. |
|
|
| --- |
|
|
| ## ✨ What's Been Built |
|
|
| ### 🖥️ Full-Stack Application |
|
|
| | Layer | Stack | |
| |---|---| |
| | **Backend API** | FastAPI (async) + Python 3.12 | |
| | **Task Queue** | Celery + Redis | |
| | **Graph + Vector DB** | Neo4j 5.x (unified) | |
| | **LLM Layer** | OpenAI, Anthropic, Google Gemini, Ollama | |
| | **Frontend** | React 18 + TypeScript + Vite | |
| | **Unified Start** | `npm run rag` (concurrently launches all 3 processes) | |
|
|
| --- |
|
|
| ## 🚀 Features |
|
|
| ### 📥 Document Ingestion Pipeline |
|
|
| - **Multi-format ingestion**: PDF, TXT, MD, DOCX, CSV, XLSX, PPTX, JSON |
| - **Web scraping**: Single-page scrape via `POST /api/documents/scrape` |
| - **Deep web crawling**: Multi-depth Playwright-powered crawler (`POST /api/documents/crawl`) via Crawl4AI |
| - **Async Celery workers**: Upload returns instantly with a `task_id`; background workers build the graph |
| - **Re-ingest**: Admin can trigger re-processing of any stored document |
| - **Document preview & download**: In-browser preview of text/Markdown; PDF download via API |
|
|
| ### 🔭 Ontology Management |
|
|
| - **Auto-generation**: LLM analyzes document chunks to propose entity types & relationship types |
| - **LLM-powered refinement**: `POST /api/ontology/refine` — refine schema with optional human feedback |
| - **Versioning**: Each schema change bumps the version (`v1.0` → `v1.1`, etc.) |
| - **Document-scoped stats**: `/api/ontology/stats?document_id=...` returns entity/relationship breakdowns for a specific document |
| - **Visual editor**: Ontology view in UI with editable entity types and relationship types |
| - **Ontology Drift Detection**: Automated drift detection compares live graph against new chunk samples; exposes pending/approved/rejected drift reports with admin approve/reject workflow |
|
|
| ### 🤖 Agentic Retrieval System |
|
|
| - **LangGraph orchestration**: State-machine ReACT agent with multi-step reasoning and fallback mechanisms |
| - **Tool routing**: Dynamically selects from Vector Search, Graph Traversal, Cypher Generation, Metadata Filtering, Community Search, and Temporal Queries |
| - **Streaming responses**: Server-Sent Events (SSE) with real-time reasoning steps surfaced in the UI |
| - **Multi-turn conversations**: Persistent conversation threads stored in Neo4j, per-user |
| - **Document-scoped queries**: Filter retrieval to a specific document via `document_id` |
| - **Graph of Thoughts (GoT)**: Optional GoT reasoning mode for complex multi-hop queries |
| - **LLM-as-a-Judge (inline)**: Optional per-response quality scoring with hallucination risk, grounded/ungrounded claims, and confidence reasoning displayed in chat |
| - **Confidence display**: Confidence score, hallucination risk, and judge reasoning shown directly in the chat bubble |
|
|
| ### 📊 RAGAS Evaluation & Quality Dashboard |
|
|
| - **`POST /api/eval/score`**: Run RAGAS-style evaluation on any Q&A pair (faithfulness, relevancy, context precision, hallucination detection) |
| - **`GET /api/eval/dashboard`**: Aggregate evaluation history — avg scores, hallucination rate, trend timeline |
| - Results persisted in Neo4j for longitudinal quality tracking |
|
|
| ### 🗺️ Graph Intelligence |
|
|
| - **D3 force-directed visualization**: Interactive knowledge graph with zoom, pan, node selection, and a details modal |
| - **Graph Export**: Export full or document-scoped graph as JSON, Cypher, or GraphML |
| - **Community Detection**: Weakly-connected-components (WCC) community assignment with `POST /api/graph/communities/assign` |
| - **Community listing**: `GET /api/graph/communities` — top communities by entity count |
| - **Temporal Queries**: `GET /api/entities/{entity_name}/at-time` — retrieve entity relationships at a historical point in time |
| - **Semantic Entity Deduplication**: Multi-stage entity resolution with configurable similarity thresholds (`POST /api/entities/deduplicate`) |
| - **Entity Enrichment**: LLM-synthesized profile summaries for every entity, stored as `e.summary` (`POST /api/entities/enrich`) |
| - **Entity Chat (scoped)**: `POST /api/entities/{entity_name}/chat` — multi-turn conversation scoped entirely to a single entity's graph neighborhood |
| - **Graph Memory Updater**: Push raw text directly into the live knowledge graph without re-ingesting a document (`POST /api/graph/update`) |
|
|
| ### 📝 Analytical Report Agent (ReACT) |
|
|
| - **`POST /api/report`**: ReACT multi-step report agent using InsightForge / PanoramaSearch / QuickSearch tools |
| - Decomposes topic into sub-questions → retrieves graph data → synthesizes sections → compiles structured markdown report |
| - Exposed in the **Insights** view (copy/download report as Markdown) |
|
|
| ### 🎭 Simulation & Persona Engine |
|
|
| - **Persona generation**: Celery task that generates personas from graph entities (`POST /api/v1/simulation/generate_personas`) |
| - **Simulation ticks**: Background tick loop (`POST /api/v1/simulation/tick`) |
| - **Live persona interview**: `POST /api/v1/simulation/interview` — roleplay chat with any graph entity injecting their Neo4j memory as system context |
| - **SimulationRunView**: Dedicated UI view for managing and interacting with simulation personas |
|
|
| ### 🛡️ Admin Dashboard |
|
|
| - **System statistics**: Node count, relationship count, LLM provider, environment |
| - **User management**: List users, update scopes/roles (RBAC) |
| - **Document vault**: View and delete all ingested documents |
| - **Graph CRUD**: Search, inspect, and delete graph nodes from the admin panel |
| - **Ontology governance**: Review and approve/reject pending ontology proposals |
| - **Celery task monitor**: View active and reserved tasks from the admin panel |
| - **Self-demotion guard**: Admins cannot demote their own account |
| - **Re-ingest button**: Re-queue any stored document from the document vault |
| - **User activity metrics**: Per-user conversation count, message count, last active timestamp |
|
|
| ### 🔐 Authentication & Security |
|
|
| - **JWT authentication**: Token-based auth with configurable expiry |
| - **RBAC scopes**: `read`, `write`, `admin` scopes enforced per endpoint |
| - **User registration**: `POST /api/auth/register` |
| - **Pydantic validation**: All API inputs validated at the model layer |
| - **Cypher injection prevention**: Schema validation and query whitelisting |
| - **File upload limits**: File size and MIME type enforcement |
|
|
| ### 🌐 Frontend (React/TypeScript) |
|
|
| Seven fully implemented views accessible from the `CORTEX` top navigation bar: |
|
|
| | Route | View | Description | |
| |---|---|---| |
| | `/` | **Home** | Animated stats dashboard — documents, entities, relationships, graph health | |
| | `/process` | **Process** | Upload files or scrape/crawl URLs; view ingestion queue and document list | |
| | `/ontology` | **Ontology** | View/edit the live ontology schema; run LLM refinement; inspect entity/relationship stats per doc | |
| | `/interact` | **Interact** | Streaming AI chat with reasoning steps, confidence, hallucination risk; conversation history | |
| | `/simulate` | **Simulate** | Simulation persona management and live interview interface | |
| | `/insights` | **Insights** | Topic-driven analytical report generation with copy/download | |
| | `/admin` | **Admin** _(admin-only)_ | Full admin panel for users, docs, tasks, ontology governance | |
|
|
| ### 🔭 Observability |
|
|
| - **OpenTelemetry**: Distributed tracing (silenced from console; configured for export) |
| - **Health check**: `GET /api/system/health` — Neo4j, Redis, Celery worker status |
| - **System stats**: `GET /api/system/stats` — document, entity, relationship, chunk counts |
| - **User stats**: `GET /api/system/my-stats` — per-user conversation and message activity |
|
|
| --- |
|
|
| ## 🏗️ Architecture |
|
|
| ``` |
| ┌─────────────────────────────────────────────────────────────────────────────┐ |
| │ React Frontend (CORTEX) │ |
| │ Home │ Process │ Ontology │ Interact │ Simulate │ Insights │ Admin │ |
| └─────────────────────────────┬───────────────────────────────────────────────┘ |
| │ HTTP / SSE |
| ┌─────────────────────────────▼───────────────────────────────────────────────┐ |
| │ FastAPI Gateway (port 8000) │ |
| │ JWT Auth · RBAC Scopes · CORS · OpenTelemetry │ |
| └──────┬──────────────────────┬──────────────────────┬────────────────────────┘ |
| │ │ │ |
| ┌──────▼──────┐ ┌───────────▼──────────┐ ┌───────▼────────────────────┐ |
| │ Ingestion │ │ ReACT Agent System │ │ Report Agent (ReACT) │ |
| │ Pipeline │ │ - Vector Search │ │ - InsightForge │ |
| │ - Parser │ │ - Graph Traversal │ │ - PanoramaSearch │ |
| │ - Ontology │ │ - Cypher Gen (GoT) │ │ - QuickSearch │ |
| │ - Extractor│ │ - Community Search │ │ - Markdown output │ |
| │ - Web │ │ - Temporal Queries │ └────────────────────────────┘ |
| │ Crawler │ │ - LLM-as-a-Judge │ |
| └──────┬──────┘ └─────────┬────────────┘ |
| │ │ |
| ┌──────▼────────────────────▼──────────────────┐ |
| │ Neo4j 5.x Database │ |
| │ Entities · Chunks · Relationships · │ |
| │ Vector Index · Conversations · │ |
| │ EvalResults · DriftReports · Users │ |
| └───────────────────────────────────────────────┘ |
| │ |
| ┌──────▼──────────────────────┐ |
| │ Celery Workers (Redis) │ |
| │ - Async document ingestion │ |
| │ - Persona generation │ |
| │ - Simulation ticks │ |
| └─────────────────────────────┘ |
| ``` |
|
|
| --- |
|
|
| ## 📦 Project Structure |
|
|
| ``` |
| graph-RAG/ |
| ├── src/graph_rag_service/ |
| │ ├── api/ |
| │ │ ├── server.py # Main FastAPI app + all API routes (1900 lines) |
| │ │ ├── auth.py # JWT auth + RBAC helpers |
| │ │ ├── admin.py # Admin sub-router |
| │ │ ├── simulation.py # Simulation / persona interview router |
| │ │ └── models.py # All Pydantic request/response models |
| │ ├── core/ |
| │ │ ├── abstractions.py # Abstract base classes (GraphStore, VectorStore, LLMProvider) |
| │ │ ├── models.py # Domain data models |
| │ │ ├── neo4j_store.py # Full Neo4j implementation (graph + vector) |
| │ │ ├── llm_factory.py # Multi-LLM provider factory + UnifiedLLMProvider |
| │ │ ├── entity_resolver.py # Semantic entity deduplication |
| │ │ └── storage.py # File storage abstraction |
| │ ├── ingestion/ |
| │ │ ├── pipeline.py # End-to-end ingestion orchestrator |
| │ │ ├── document_processor.py # Multi-format document parsing |
| │ │ ├── ontology_generator.py # LLM ontology generation + refinement |
| │ │ ├── extractor.py # Entity + relationship extraction |
| │ │ ├── web_crawler.py # Playwright-based deep web crawler (Crawl4AI) |
| │ │ └── persona_generator.py # Simulation persona generation |
| │ ├── retrieval/ |
| │ │ ├── agent.py # LangGraph ReACT retrieval agent |
| │ │ ├── tools.py # Retrieval tools + RAGEvaluator (RAGAS) |
| │ │ └── report_agent.py # ReACT analytical report agent |
| │ ├── services/ |
| │ │ ├── graph_memory_updater.py # Push raw text → live graph |
| │ │ ├── entity_enricher.py # LLM entity profile summaries |
| │ │ └── ontology_drift_detector.py # Automated schema drift detection |
| │ ├── workers/ |
| │ │ └── celery_worker.py # Celery app + ingest_document_task |
| │ ├── observability/ |
| │ │ └── tracing.py # OpenTelemetry setup (console suppressed) |
| │ ├── config.py # Pydantic settings (all env vars) |
| │ └── main.py # Uvicorn entry point |
| ├── frontend-react/ |
| │ └── src/ |
| │ ├── views/ |
| │ │ ├── Home.tsx # Animated stats dashboard |
| │ │ ├── Process.tsx # Document upload + URL scrape/crawl |
| │ │ ├── Ontology.tsx # Schema editor + stats |
| │ │ ├── InteractionView.tsx # Streaming chat + conversation history |
| │ │ ├── SimulationRunView.tsx # Persona simulation UI |
| │ │ ├── InsightsView.tsx # Report generation + copy/download |
| │ │ ├── AdminDashboard.tsx # Full admin panel |
| │ │ └── Login.tsx # Login page |
| │ ├── components/ |
| │ │ └── GraphCanvas.tsx # D3 force-directed graph + node modal |
| │ ├── context/ |
| │ │ └── AuthContext.tsx # JWT auth context + hooks |
| │ └── App.tsx # Router + top-nav (CORTEX branding) |
| ├── tests/ # Test suite |
| ├── data/uploads/ # Uploaded documents (local storage) |
| ├── .env.example # All configurable environment variables |
| ├── pyproject.toml # Python project + uv dependencies |
| ├── package.json # Unified start scripts (npm run rag) |
| ├── ARCHITECTURE.md # Detailed architecture design doc |
| └── QUICKSTART.md # 5-minute quick start guide |
| ``` |
|
|
| --- |
|
|
| ## ⚡ Quick Start |
|
|
| ### Prerequisites |
|
|
| - Python 3.12+ |
| - Node.js 18+ |
| - Neo4j 5.x (running) with **APOC** and **Graph Data Science (GDS)** plugins installed |
| - Redis (running) |
| - Ollama *(optional, for local LLMs)* |
|
|
| ### 1. Clone & Install |
|
|
| ```bash |
| git clone <repository-url> |
| cd graph-RAG |
| |
| # Installs Python deps (uv), frontend (npm), and Playwright Chromium |
| npm install |
| ``` |
|
|
| ### 2. Configure Environment |
|
|
| ```bash |
| cp .env.example .env |
| # Fill in NEO4J_URI, NEO4J_PASSWORD, and your LLM API keys |
| ``` |
|
|
| ### 3. Start Neo4j |
|
|
| ```bash |
| docker run -d --name neo4j \ |
| -p 7474:7474 -p 7687:7687 \ |
| -e NEO4J_AUTH=neo4j/password \ |
| neo4j:latest |
| ``` |
|
|
| ### 4. Start Redis |
|
|
| ```bash |
| docker run -d --name redis -p 6379:6379 redis:alpine |
| ``` |
|
|
| ### 5. Launch Everything |
|
|
| ```bash |
| npm run rag |
| ``` |
|
|
| This starts three color-coded processes concurrently: |
|
|
| | Process | URL | |
| |---|---| |
| | **API Server** | `http://localhost:8000` | |
| | **API Docs** | `http://localhost:8000/docs` | |
| | **React Frontend** | `http://localhost:5173` | |
|
|
| > Default credentials: `admin` / `admin` |
|
|
| --- |
|
|
| ## 🔑 Environment Variables |
|
|
| Copy `.env.example` to `.env` and configure: |
|
|
| ```env |
| # Neo4j |
| NEO4J_URI=bolt://localhost:7687 |
| NEO4J_USER=neo4j |
| NEO4J_PASSWORD=password |
| |
| # Redis |
| REDIS_HOST=localhost |
| REDIS_PORT=6379 |
| |
| # LLM Provider (openai | anthropic | gemini | ollama) |
| DEFAULT_LLM_PROVIDER=gemini |
| GOOGLE_API_KEY=your-key-here |
| |
| # Optional: OpenAI / Anthropic |
| OPENAI_API_KEY=sk-... |
| ANTHROPIC_API_KEY=sk-ant-... |
| |
| # Optional: Ollama (local) |
| OLLAMA_BASE_URL=http://localhost:11434 |
| OLLAMA_MODEL=deepseek-r1:7b |
| OLLAMA_EMBEDDING_MODEL=nomic-embed-text |
| |
| # Feature flags |
| ENABLE_LLM_JUDGE=true |
| |
| # Security |
| SECRET_KEY=change-this-in-production |
| ACCESS_TOKEN_EXPIRE_MINUTES=1440 |
| ``` |
|
|
| --- |
|
|
| ## 🌐 API Reference |
|
|
| ### Authentication |
| | Method | Endpoint | Description | |
| |---|---|---| |
| | `POST` | `/api/auth/register` | Register new user | |
| | `POST` | `/api/auth/login` | Login → JWT token | |
| | `GET` | `/api/auth/me` | Get current user info | |
|
|
| ### Documents |
| | Method | Endpoint | Description | |
| |---|---|---| |
| | `POST` | `/api/documents/upload` | Upload file (PDF, DOCX, TXT, MD, CSV, XLSX, PPTX, JSON) | |
| | `POST` | `/api/documents/scrape` | Scrape single URL → ingest | |
| | `POST` | `/api/documents/crawl` | Deep multi-page Playwright crawl → ingest *(API Only)* | |
| | `GET` | `/api/documents` | List all ingested documents | |
| | `DELETE` | `/api/documents/{id}` | Delete document + graph chunks | |
| | `GET` | `/api/documents/{id}/download` | Download source file | |
| | `GET` | `/api/documents/{id}/preview` | Preview text content | |
| | `GET` | `/api/documents/status/{task_id}` | Ingestion task status | |
|
|
| ### Query & Chat |
| | Method | Endpoint | Description | |
| |---|---|---| |
| | `POST` | `/api/query` | Agentic query (streaming or JSON); supports `document_id`, `use_got` | |
| | `GET` | `/api/conversations` | List conversation threads | |
| | `GET` | `/api/conversations/{id}` | Get conversation + messages | |
| | `DELETE` | `/api/conversations/{id}` | Delete conversation | |
|
|
| ### Ontology |
| | Method | Endpoint | Description | |
| |---|---|---| |
| | `GET` | `/api/ontology` | Get current ontology | |
| | `PUT` | `/api/ontology` | Update ontology (admin) | |
| | `POST` | `/api/ontology/refine` | LLM-powered ontology refinement | |
| | `GET` | `/api/ontology/stats` | Entity/relationship counts (optional doc filter) | |
| | `POST` | `/api/ontology/drift/detect` | Trigger drift detection | |
| | `GET` | `/api/ontology/drift` | List drift reports | |
| | `POST` | `/api/ontology/drift/{id}/approve` | Approve drift → merge into ontology | |
| | `POST` | `/api/ontology/drift/{id}/reject` | Reject drift report | |
|
|
| ### Graph |
| | Method | Endpoint | Description | |
| |---|---|---| |
| | `GET` | `/api/graph/visualization` | Graph nodes + edges for D3 rendering | |
| | `GET` | `/api/graph/export` | Export graph (json \| cypher \| graphml) | |
| | `POST` | `/api/graph/update` | Push raw text → merge into live graph | |
| | `POST` | `/api/graph/communities/assign` | Run WCC community detection | |
| | `GET` | `/api/graph/communities` | List top communities | |
|
|
| ### Entities |
| | Method | Endpoint | Description | |
| |---|---|---| |
| | `POST` | `/api/entities/deduplicate` | Semantic entity resolution + merge | |
| | `POST` | `/api/entities/enrich` | Generate LLM summaries for all entities | |
| | `GET` | `/api/entities/{name}/summary` | Get enriched entity profile | |
| | `POST` | `/api/entities/{name}/chat` | Multi-turn entity-scoped chat | |
| | `GET` | `/api/entities/{name}/at-time` | Temporal query (ISO 8601 date) | |
|
|
| ### Reports & Evaluation |
| | Method | Endpoint | Description | |
| |---|---|---| |
| | `POST` | `/api/report` | Generate ReACT analytical report (markdown) | |
| | `POST` | `/api/eval/score` | RAGAS evaluation of a Q&A pair | |
| | `GET` | `/api/eval/dashboard` | Evaluation history dashboard | |
|
|
| ### Simulation |
| | Method | Endpoint | Description | |
| |---|---|---| |
| | `POST` | `/api/v1/simulation/interview` | Live persona interview (in-character LLM) | |
| | `GET` | `/api/v1/simulation/report` | Sandbox analytical report *(API Only)* | |
| | `POST` | `/api/v1/simulation/generate_personas` | Queue persona generation task *(API Only)* | |
| | `POST` | `/api/v1/simulation/tick` | Advance simulation tick *(API Only)* | |
|
|
| ### System & Admin |
| | Method | Endpoint | Description | |
| |---|---|---| |
| | `GET` | `/api/system/health` | Neo4j + Redis + Celery health | |
| | `GET` | `/api/system/stats` | Document, entity, relationship counts | |
| | `GET` | `/api/system/my-stats` | Current user's activity stats | |
| | `GET` | `/api/system/formats` | Supported ingestion file formats | |
| | `GET` | `/api/admin/stats` | Admin-only system stats | |
| | `GET` | `/api/admin/users` | List all users | |
| | `PUT` | `/api/admin/users/{username}/role` | Update user scopes | |
| | `GET` | `/api/admin/tasks` | View Celery tasks | |
| | `GET` | `/api/admin/documents` | Admin document vault | |
| | `POST` | `/api/admin/documents/{id}/reingest` | Re-queue document for ingestion | |
| | `GET` | `/api/admin/graph/nodes` | Search graph nodes | |
| | `DELETE` | `/api/admin/graph/nodes/{id}` | Delete a graph node | |
|
|
| --- |
|
|
| ## 🧪 Testing |
|
|
| ```bash |
| # Run tests |
| uv run pytest |
| |
| # With coverage |
| uv run pytest --cov=src/graph_rag_service |
| ``` |
|
|
| --- |
|
|
| ## 🚀 Production Deployment |
|
|
| | Process | Command | |
| |---|---| |
| | **API Server** | `uv run python main.py` | |
| | **Celery Worker** | `uv run celery -A src.graph_rag_service.workers.celery_worker worker --loglevel=info --concurrency=4 --pool=threads` | |
| | **React Build** | `cd frontend-react && npm run build` | |
|
|
| The built React assets can be served directly by FastAPI (static file mount), or deployed to a CDN separately. Neo4j and Redis can be run via Docker, managed cloud services (AuraDB, Redis Cloud), or self-hosted. |
|
|
| --- |
|
|
| ## 📄 Additional Documentation |
|
|
| - **[ARCHITECTURE.md](./ARCHITECTURE.md)** — Deep dive into the system design, data flow, and component interactions |
| - **[QUICKSTART.md](./QUICKSTART.md)** — 5-minute environment setup guide |
| - **`/docs`** — Interactive Swagger UI (auto-generated from FastAPI) |
|
|
| --- |
|
|
| **Project Status**: Production-grade MVP · Actively developed |
| **License**: Proprietary — all rights reserved |
|
|
