v4.0: Update README with OCR, Chatbot, and Redlining docs

README.md

@@ -10,9 +10,17 @@ app_file: app.py
 pinned: false
 ---
 
-# 🛡️ ClauseGuard — World's Best Open-Source Legal Contract Analysis
+# 🛡️ ClauseGuard v4.0 — World's Best Open-Source Legal Contract Analysis
 
-**ClauseGuard** is the most comprehensive open-source AI-powered legal contract analysis tool. It analyzes contracts using state-of-the-art legal NLP models and provides actionable risk assessments.
+**ClauseGuard** is the most comprehensive open-source AI-powered legal contract analysis tool. It analyzes contracts using state-of-the-art legal NLP models and provides actionable risk assessments, Q&A chatbot, clause redlining, and OCR for scanned PDFs.
+
+## 🆕 What's New in v4.0
+
+| Feature | Description |
+|---------|-------------|
+| **🔍 OCR for Scanned PDFs** | Smart PDF router: auto-detects native vs scanned PDFs. Scanned PDFs are processed via docTR OCR engine (CPU-friendly, ~150MB models) |
+| **💬 Contract Q&A Chatbot** | RAG-powered chatbot that answers questions about your analyzed contract. Uses sentence-transformers for retrieval + Qwen2.5-7B via HF Inference API for generation |
+| **✏️ Clause Redlining** | 3-tier system: (1) Template lookup from 18+ legal templates based on FTC/EU standards, (2) Keyword-based matching, (3) LLM refinement for CRITICAL/HIGH risk clauses |
 
 ## ✨ Core Features
 
@@ -26,9 +34,12 @@ pinned: false
 | **Obligation Tracker** | Categorizes action items: monetary 💰, compliance ⚖️, reporting 📊, delivery 📦, termination 🛑 |
 | **Compliance Checker** | Validates against GDPR, CCPA, SOX, HIPAA, and FINRA requirements |
 | **Contract Comparison** | Side-by-side diff between two contracts with alignment scoring |
+| **Clause Redlining** | Suggests safer alternatives for risky clauses with legal citations |
+| **Q&A Chatbot** | Ask questions about your contract using RAG (Retrieval-Augmented Generation) |
+| **OCR Support** | Process scanned PDFs with docTR OCR engine |
 
 ### Document Support
-- **PDF** parsing via `pdfplumber`
+- **PDF** parsing via `pdfplumber` (native) + `docTR` OCR (scanned)
 - **DOCX/DOC** parsing via `python-docx`
 - **TXT / Markdown** direct text input
 
@@ -36,6 +47,8 @@ pinned: false
 - **3-Panel Professional Layout** — Upload sidebar + Main analysis + Summary dashboard
 - **Document Viewer** — Inline entity highlights (colored annotations)
 - **Clause Cards** — Expandable risk-badged cards with confidence scores
+- **Redlining Tab** — Side-by-side original vs suggested safer alternatives
+- **Q&A Chat Tab** — Conversational interface to ask questions about the contract
 - **Export Reports** — JSON (structured) and CSV (tabular) downloads
 - **Color-Coded Risk Badges** — Instant visual triage
 
@@ -44,12 +57,61 @@ pinned: false
 | Component | Technology |
 |-----------|------------|
 | Clause Classification | `Mokshith31/legalbert-contract-clause-classification` — LoRA adapter on `nlpaueb/legal-bert-base-uncased`, fine-tuned on CUAD 41-class taxonomy |
-| NER | Rule-based with 7 entity types (dates, money, parties, jurisdictions, defined terms) |
-| NLI | Heuristic contradiction detection with 5 conflict patterns + missing-clause detection |
+| Legal NER | `matterstack/legal-bert-ner` (ML) with regex fallback for 7 entity types |
+| NLI | `cross-encoder/nli-deberta-v3-base` (semantic contradiction detection) |
+| Embeddings | `sentence-transformers/all-MiniLM-L6-v2` (384-dim, RAG retrieval) |
+| LLM | `Qwen/Qwen2.5-7B-Instruct` via HF Inference API (chatbot + redlining) |
+| OCR | `docTR` (fast_base + crnn_vgg16_bn) for scanned PDF text extraction |
 | Compliance | Regulatory keyword matching across GDPR, CCPA, SOX, HIPAA, FINRA |
-| Comparison | SequenceMatcher-based clause alignment with risk delta analysis |
+| Comparison | Semantic similarity with sentence embeddings + string matching fallback |
 | Obligations | Regex pattern matching across 5 obligation categories |
 
+## 🔍 OCR Architecture (Smart PDF Router)
+
+```
+PDF uploaded
+    ↓
+[detect_if_scanned] — pdfplumber extracts >50 chars/page?
+    ↓                           ↓
+  Native PDF               Scanned PDF
+    ↓                           ↓
+  pdfplumber              docTR OCR (CPU)
+    ↓                           ↓
+  Contract text → existing analysis pipeline
+```
+
+## 💬 Q&A Chatbot Architecture (RAG)
+
+```
+User asks question about their contract
+        ↓
+[1] Embed question with all-MiniLM-L6-v2
+        ↓
+[2] Retrieve top-5 most relevant chunks from contract
+        ↓
+[3] Build prompt:
+    - System: ClauseGuard analysis results (clauses, entities, risk scores)
+    - Context: Retrieved contract chunks (≤2.5K tokens)
+    - User question
+        ↓
+[4] Stream response from Qwen2.5-7B via HF Inference API
+```
+
+**Key design:** Analyzed data (clauses, entities, risk scores) goes in the system prompt — NOT through RAG retrieval. Only the raw contract text goes through RAG. This gives the model both structured analysis AND verbatim evidence.
+
+## ✏️ Clause Redlining Architecture (3-Tier)
+
+| Tier | Method | Speed | Hallucination Risk |
+|------|--------|-------|--------------------|
+| **1. Template Lookup** | 18+ pre-written safe alternatives based on FTC/EU/CFPB standards | Instant | Zero |
+| **2. Keyword Matching** | Match clause text to relevant templates via legal keywords | Instant | Zero |
+| **3. LLM Refinement** | Qwen2.5-7B adapts template to specific clause context | ~3-5s | Low (template-anchored) |
+
+Anti-hallucination guardrails:
+- **Template anchor:** LLM can only refine, not generate from scratch
+- **Legal citation:** Every suggestion includes legal basis and consumer standard
+- **Disclaimer:** Clear "Not legal advice" warning
+
 ## 📊 Risk Scoring Methodology
 
 Risk scores combine clause detection with weighted severity:
@@ -65,16 +127,10 @@ Final score normalized to 0-100 with letter grades:
 - D (50-69): High risk
 - F (70+): Critical risk
 
-## 📚 Datasets & Research
-
-- [CUAD](https://huggingface.co/datasets/theatticusproject/cuad-qa) — 510 contracts, 13K annotations, 41 clause categories
-- [LegalBench](https://huggingface.co/datasets/nguha/legalbench) — 322 legal reasoning tasks
-- [LexGLUE](https://huggingface.co/datasets/coastalcph/lex_glue) — Unfair Terms of Service classification
-- Paper: [CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review](https://arxiv.org/abs/2103.06268) (Hendrycks et al., 2021)
-
 ## 🚀 Usage
 
 1. **Upload** a contract (PDF, DOCX, or TXT) or paste text directly
+   - 💡 Scanned PDFs are automatically processed with OCR
 2. Click **Analyze Contract**
 3. View results across tabs:
    - **Document**: Full text with inline entity highlights
@@ -83,7 +139,9 @@ Final score normalized to 0-100 with letter grades:
    - **Contradictions**: Conflicting clauses and missing provisions
    - **Obligations**: Action items categorized by type
    - **Compliance**: Regulatory framework checks
+   - **Redlining**: ✏️ Safer clause alternatives with legal citations
 4. **Export** JSON/CSV reports
+5. Switch to **💬 Contract Q&A** tab to ask questions about your contract
 
 ## 🔀 Compare Contracts
 
@@ -91,7 +149,6 @@ Switch to the **Compare Contracts** tab to:
 - Upload or paste two contracts side-by-side
 - See clause-level diffs (added, removed, modified)
 - Get an alignment score and risk delta
-- View raw JSON comparison data
 
 ## ⚠️ Disclaimer
 
@@ -103,6 +160,8 @@ Switch to the **Compare Contracts** tab to:
 - [Clause Classifier Model](https://huggingface.co/Mokshith31/legalbert-contract-clause-classification)
 - [Legal-BERT Base](https://huggingface.co/nlpaueb/legal-bert-base-uncased)
 - [CUAD Dataset](https://huggingface.co/datasets/theatticusproject/cuad-qa)
+- [Qwen2.5-7B (Chatbot LLM)](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct)
+- [docTR OCR](https://github.com/mindee/doctr)
 - [CUAD Paper (arXiv:2103.06268)](https://arxiv.org/abs/2103.06268)
 
 ---