---
license: mit
language:
- en
base_model:
- Qwen/Qwen3.6-27B
- Qwen/Qwen3.6-35B-A3B
pipeline_tag: image-to-text
tags:
- medical
---
<div align="center">

<img src="https://img.shields.io/badge/AMD_Instinct-MI300X-ED1C24?style=for-the-badge&logo=amd&logoColor=white" />
<img src="https://img.shields.io/badge/ROCm-Stack-ED1C24?style=for-the-badge&logo=amd&logoColor=white" />
<img src="https://img.shields.io/badge/vLLM-Inference-6D28D9?style=for-the-badge" />
<img src="https://img.shields.io/badge/Qwen-Multimodal-0EA5E9?style=for-the-badge" />
<img src="https://img.shields.io/badge/FastAPI-0.115-009688?style=for-the-badge&logo=fastapi&logoColor=white" />
<img src="https://img.shields.io/badge/Python-3.12+-3776AB?style=for-the-badge&logo=python&logoColor=white" />

<br /><br />

# 🏥 MediAgent

### Autonomous Multi-Agent Medical Imaging Analysis System

**Five specialized AI agents. One radiological verdict. Running entirely on AMD.**

*AMD Developer Hackathon 2026 · Track: Vision & Multimodal AI*

<br />

> Built by **Ramyar** — Security researcher & full-stack developer, Sulaymaniyah, Iraq

</div>

---

## What Is MediAgent?

MediAgent is a production-grade autonomous AI system that analyzes medical images — X-rays, MRI scans, CT scans — through a five-agent pipeline and generates structured, peer-reviewed clinical radiology reports in real time.

Upload an image. Watch five AI agents execute live. Get a formal radiology report with differential diagnoses, ICD-10 codes, a quality score, and a FHIR R4 export ready for any EMR system.

**No cloud APIs. No OpenAI. No Nvidia.**
Pure AMD MI300X inference. Local. Private. Fast.

---

## The Pipeline

```
┌─────────────────────────────────────────────────────────────────────┐
│                        IMAGE UPLOAD                                 │
│              PNG / JPG / DICOM (.dcm) — up to 20 MB                │
└──────────────────────────┬──────────────────────────────────────────┘
                           │
          ┌────────────────┴────────────────┐
          │         PARALLEL STAGE          │
          ▼                                 ▼
┌─────────────────┐               ┌─────────────────┐
│  INTAKE AGENT   │               │  VISION AGENT   │
│                 │               │                 │
│ • Validates     │               │ • Multimodal    │
│   image payload │               │   Qwen analysis │
│ • Normalizes    │               │ • Anatomical    │
│   clinical text │               │   findings      │
│ • Extracts      │               │ • Severity per  │
│   demographics  │               │   region        │
│ • Safety triage │               │ • Confidence    │
│   (16 keywords) │               │   scoring       │
│ • Modality hint │               │ • Anomaly flags │
└────────┬────────┘               └────────┬────────┘
         └──────────────┬──────────────────┘
                        │
                        ▼
            ┌───────────────────────┐
            │    RESEARCH AGENT     │
            │                       │
            │ • KB cross-reference  │
            │   (15 conditions)     │
            │ • Demographic weight  │
            │ • Ranked differentials│
            │ • ICD-10 codes        │
            │ • Match probabilities │
            └───────────┬───────────┘
                        │
                        ▼
            ┌───────────────────────┐
            │     REPORT AGENT      │
            │                       │
            │ • ACR/NICE format     │
            │ • Clinical history    │
            │ • Technique section   │
            │ • Findings narrative  │
            │ • Impression + top Dx │
            │ • Recommendations     │
            └───────────┬───────────┘
                        │
                        ▼
            ┌───────────────────────┐
            │     CRITIC AGENT      │
            │                       │
            │ • Cross-validates     │
            │   report vs findings  │
            │ • Quality score 0-100 │
            │ • Uncertainty flags   │
            │ • Disclaimer enforce  │
            └───────────┬───────────┘
                        │
                        ▼
┌─────────────────────────────────────────────────────────────────────┐
│                      FINAL REPORT                                   │
│         Structured JSON · PDF Export · FHIR R4 DiagnosticReport    │
└─────────────────────────────────────────────────────────────────────┘
```

INTAKE and VISION execute **concurrently** — cutting wall-clock latency by running the two most expensive operations in parallel. Everything downstream sequences after both complete.

---

## AMD Hardware Stack

| Component | Technology |
|---|---|
| **GPU** | AMD Instinct MI300X |
| **GPU Software** | ROCm — AMD's open-source GPU compute platform |
| **Inference Server** | vLLM (ROCm build) at `localhost:8000/v1` |
| **Model** | Qwen multimodal — native vision + text |
| **Backend** | FastAPI 0.115 + Uvicorn |
| **Frontend** | Vanilla JS + Tailwind CSS + SSE streaming |

This project is a direct proof of concept that AMD's ROCm stack is **production-viable for real-world medical AI**. Every inference call — vision analysis, clinical normalization, report synthesis, peer review, post-report chat — runs on AMD MI300X. Zero CUDA dependency. Zero cloud API calls.

---

## Key Features

### 🔴 Real-Time SSE Streaming
Watch the pipeline execute live, agent by agent. Every status transition — WAITING → RUNNING → DONE — streams to the dashboard as it happens via Server-Sent Events. Per-agent runtime counters track exactly how long each step takes.

### 👁️ Multimodal Vision Analysis
Qwen processes the raw medical image natively. It returns structured JSON: detected modality, technical quality assessment, per-region findings with anatomical names, radiological descriptions, severity levels (NORMAL / INCIDENTAL / SIGNIFICANT / CRITICAL), confidence scores (0–100), and anomaly flags.

### 🔬 Medical Knowledge Base + ICD-10 Mapping
The Research Agent cross-references vision findings against 15 curated clinical conditions spanning pulmonary, neurological, abdominal, musculoskeletal, and vascular pathology. Every differential diagnosis comes with an ICD-10 code, match probability, and a sentence explaining exactly why the condition matches the findings.

### 🛡️ Critic Agent QA
Every report goes through a peer-review pass before delivery. The Critic checks that all anomalies from the Vision Agent appear in the report, flags low-confidence findings, assigns a quality score (completeness 30% + accuracy 40% + safety 20% + compliance 10%), and hard-caps the score at 40/100 if a core agent failed.

### 🏥 DICOM Support
Upload real `.dcm` files. MediAgent extracts 20+ metadata fields — patient name, study date, institution, modality, body part, KVP, slice thickness, pixel spacing, image dimensions — and pre-populates the intake form automatically. MONOCHROME1 inversion and multi-frame handling included.

### 📋 FHIR R4 Export
Every report can be exported as a fully conformant HL7 FHIR R4 DiagnosticReport resource. Includes an inline Patient resource, Observation resources, LOINC and SNOMED CT codes, severity mapping, full report text in `presentedForm`, and custom extensions for AI quality score and pipeline status. Ready to import into Epic, Cerner, or any FHIR-capable EMR.

### 💬 Post-Report Clinical Chat
After the report is delivered, a ClinicalAdvisorAgent is available for follow-up questions. It answers in 2–4 sentences with direct reference to the report findings. Qwen's thinking/reasoning mode is explicitly disabled — answers are fast, direct, and clinical.

### 🔒 Hard Safety Enforcement
- **16 deterministic safety keywords** — chest pain, stroke symptoms, acute trauma, hemoptysis, sepsis, spinal trauma, and more — trigger urgent flags regardless of LLM output.
- **Age-based alerts** — pediatric (<18) and geriatric (>75) cases are automatically flagged for expert review.
- **Mandatory AI disclaimer** — enforced at two independent layers (Report Agent + Critic Agent) and cannot be bypassed or modified by the LLM.
- **Graceful degradation** — the pipeline produces a report even if individual agents fail, always marking what succeeded and what didn't.

### 📄 Client-Side PDF Export
Full radiology report exported as a formatted PDF directly in the browser using jsPDF — severity color banner, all six report sections, DICOM metadata, QA score. No server round-trip needed.

---

## Agent Architecture

### IntakeAgent
Validates the image payload (minimum size, valid base64), applies deterministic safety triage, and normalizes clinical language. For simple inputs under 120 characters it skips the LLM entirely and uses a built-in layman-to-medical term map (22 entries: "can't breathe" → "dyspnea", "lump" → "mass/nodule", "dizzy" → "dizziness/vertigo", etc.). Only calls the LLM for complex clinical narratives with comorbidities or medical history. Falls back cleanly to raw input preservation if the LLM is unavailable.

### VisionAgent
Sends the base64 image and clinical context to Qwen at temperature 0.0 with a strict JSON schema enforced via system prompt. Handles malformed enum values from the LLM with safe conversion fallbacks — a single bad field never drops a finding. Tracks token usage and anomaly counts in the output metadata.

### ResearchAgent
Pre-filters the knowledge base to only conditions compatible with the detected modality before sending to the LLM — reducing prompt size and improving accuracy. Enforces strict output rules: only conditions from the KB, 2–4 differentials maximum, 5% minimum probability, exact ICD-10 codes, and evidence sentences that actually explain the match.

### ReportAgent
Builds a structured prompt with clearly labeled sections — clinical history, imaging technique, findings block, differentials block — and asks the LLM to synthesize them into a formal ACR/NICE radiology report. The disclaimer is overwritten to the exact regulatory string after LLM generation, unconditionally.

### CriticAgent
Operates at temperature 0.0 for fully deterministic QA. Receives the draft report and the full pipeline state including raw vision findings. Checks every anomaly is accounted for, flags low-confidence observations, and appends a `[QUALITY ASSESSMENT]` block to the recommendations section with score, issues, and uncertainty warnings.

### ClinicalAdvisorAgent
Activated only after report delivery, scoped to the specific report's findings. Strips all Qwen thinking output via multi-layer regex before returning the answer — handles `<think>` XML blocks, markdown think fences, and plain-text reasoning preambles.

---

## LLM Client

The `LLMClient` wraps the OpenAI Python SDK pointed at the local vLLM endpoint. It handles:

- Text completions with optional JSON mode enforcement
- Multimodal completions with base64 image injection
- Token-level streaming with an `on_token` callback
- 3-attempt retry loop with 1-second flat backoff
- 90-second timeout per call
- Dual-strategy JSON extraction: direct parse first, then character-by-character brace-matching fallback for responses where the LLM adds conversational padding

---

## Medical Knowledge Base

15 conditions covering the most common radiological findings across all supported modalities:

| Condition | ICD-10 | Modalities | Severity |
|---|---|---|---|
| Community-Acquired Pneumonia | J18.9 | X-RAY, CT | SIGNIFICANT |
| Cardiogenic Pulmonary Edema | J81.0 | X-RAY, CT | CRITICAL |
| Pleural Effusion | J90 | X-RAY, CT, MRI | SIGNIFICANT |
| Spontaneous Pneumothorax | J93.9 | X-RAY, CT | CRITICAL |
| Intracerebral Hemorrhage | I61.9 | CT, MRI | CRITICAL |
| Ischemic Stroke | I63.9 | CT, MRI | CRITICAL |
| Intracranial Neoplasm | C71.9 | MRI, CT | SIGNIFICANT |
| Abdominal Aortic Aneurysm | I71.4 | CT, MRI | CRITICAL |
| Nephrolithiasis | N20.0 | CT, X-RAY | SIGNIFICANT |
| Small Bowel Obstruction | K56.6 | X-RAY, CT | SIGNIFICANT |
| Long Bone Fracture | S82.902 | X-RAY, CT | SIGNIFICANT |
| Degenerative Joint Disease | M19.90 | X-RAY, MRI | INCIDENTAL |
| Hepatic Steatosis | K76.0 | CT, MRI | INCIDENTAL |
| Herniated Disc | M51.16 | MRI, CT | SIGNIFICANT |
| Pulmonary Nodule | R91.1 | X-RAY, CT | SIGNIFICANT |

---

## API Reference

| Method | Endpoint | Description |
|---|---|---|
| `GET` | `/` | Clinical dashboard UI |
| `GET` | `/health` | System health, version, active sessions |
| `GET` | `/metrics/gpu` | Live AMD GPU metrics (util, VRAM, temp, power) |
| `POST` | `/analyze` | Synchronous pipeline → full JSON report |
| `POST` | `/analyze/stream` | Real-time SSE streaming pipeline |
| `GET` | `/status/{report_id}` | Poll live pipeline state |
| `POST` | `/chat/{report_id}` | Post-report clinical Q&A |
| `GET` | `/api/docs` | Swagger UI |
| `GET` | `/api/redoc` | ReDoc UI |

### `/analyze/stream` — SSE Event Types

```json
// Agent status update (emitted on every state transition)
{"agent": "VISION", "status": "RUNNING"}
{"agent": "VISION", "status": "DONE"}

// Final report (emitted when pipeline completes)
{"type": "report", "data": {...}, "report_id": "REP-A3F9C2D1B4E7"}

// Error
{"type": "error", "message": "Pipeline produced no report"}
```

### Form Fields (`/analyze`, `/analyze/stream`)

| Field | Type | Required | Notes |
|---|---|---|---|
| `image` | File | ✅ | PNG, JPG, or DICOM (.dcm), max 20 MB |
| `symptoms` | string | — | Free-text chief complaint |
| `age` | integer | — | 0–120 |
| `sex` | string | — | `M`, `F`, or `O` |
| `clinical_context` | string | — | Medical history, referral details |

---

## Data Models

```
PatientInput
    └── image_base64, symptoms, age, sex, clinical_context

PipelineState
    ├── agent_statuses: {INTAKE, VISION, RESEARCH, REPORT, CRITIC}
    ├── intake_output: IntakeOutput
    ├── vision_output: VisionOutput
    │       └── findings: [VisionFinding, ...]
    │               └── anatomical_region, description, severity,
    │                   confidence, confidence_score, is_anomaly
    ├── research_output: ResearchOutput
    │       └── differential_diagnoses: [KnowledgeMatch, ...]
    │               └── condition_name, match_probability,
    │                   supporting_evidence, differential_rank, icd10_code
    ├── report_draft: ReportSection
    │       └── clinical_history, technique, findings, impression,
    │           recommendations, disclaimer
    └── final_report: FinalReport
            └── report_id, patient_metadata, sections, vision_summary,
                research_summary, overall_severity, agent_pipeline_status,
                generation_timestamp
```

---

## Project Structure

```
mediagent/
├── main.py                  ← FastAPI server, all routes, SSE orchestration
├── core/
│   ├── llm.py               ← LLM client (retry, vision, streaming, JSON extraction)
│   ├── models.py            ← All Pydantic v2 data models
│   ├── pipeline.py          ← Parallel pipeline orchestrator
│   ├── dicom.py             ← DICOM parser (pydicom + numpy + Pillow)
│   └── fhir.py              ← FHIR R4 DiagnosticReport builder
├── agents/
│   ├── intake.py            ← Input validation, normalization, safety triage
│   ├── vision.py            ← Multimodal image analysis
│   ├── research.py          ← KB matching, ICD-10, differential diagnosis
│   ├── report.py            ← ACR/NICE radiology report synthesis
│   ├── critic.py            ← QA validation, quality scoring
│   └── advisor.py           ← Post-report clinical Q&A
├── static/
│   └── index.html           ← Full dashboard (Tailwind + Chart.js + SSE)
├── requirements.txt
└── .env.example
```

---

## Getting Started

### Prerequisites

- Python 3.12+
- vLLM running a Qwen multimodal model on ROCm, accessible at `http://localhost:8000/v1`
- ROCm-compatible AMD GPU (MI300X recommended)

### Installation

```bash
# Clone the repository
git clone https://github.com/Ramyar2007/mediagent
cd mediagent

# Install Python dependencies
pip install -r requirements.txt

# Configure environment
cp .env.example .env
# Edit .env and set LLM_BASE_URL to your vLLM endpoint
```

### Environment Variables

```env
LLM_BASE_URL=http://localhost:8000/v1   # vLLM OpenAI-compatible endpoint
LLM_MODEL=/model                         # Model path served by vLLM
APP_PORT=8090                            # Server port
```

### Run

```bash
python main.py
```

Dashboard available at **http://localhost:8090**

Swagger docs at **http://localhost:8090/api/docs**

---

## Dependencies

| Package | Version | Purpose |
|---|---|---|
| `fastapi` | 0.115.6 | Web framework |
| `uvicorn[standard]` | 0.34.0 | ASGI server |
| `openai` | 1.58.1 | SDK for vLLM OpenAI-compatible API |
| `python-multipart` | 0.0.20 | Multipart form / file upload |
| `pydantic` | 2.10.5 | Data validation and serialization |
| `Pillow` | 11.1.0 | Image processing for DICOM conversion |
| `pydicom` | 2.4.4 | DICOM file parsing and metadata extraction |
| `numpy` | 1.26.4 | Pixel array normalization for DICOM |

Optional: `amdsmi` Python library — used automatically when available for more accurate GPU metrics than the `rocm-smi` CLI fallback.

---

## Clinical Safety

MediAgent is built with clinical safety as a first-class concern, not an afterthought.

**Mandatory disclaimer** — enforced at two independent code layers and cannot be overridden by any LLM output:

> *"This analysis is AI-generated and must be reviewed by a licensed radiologist before any clinical decisions are made."*

**Hard safety rules that run deterministically, without LLM involvement:**
- 16 urgent clinical keywords trigger immediate flags before any AI processing
- Pediatric and geriatric age thresholds auto-flag for specialist review
- Quality score is hard-capped at 40/100 if core agents (Vision, Report) fail
- Low-confidence findings are always flagged with confirmatory imaging recommendations
- The disclaimer is re-enforced after every LLM call, unconditionally

**This system is a decision support tool, not a clinical decision maker.** Every output is intended to assist, not replace, a licensed radiologist.

---

## Dashboard Preview

The single-page clinical dashboard provides:

- **Live pipeline panel** — real-time agent status cards with per-step runtime counters
- **Analytics tab** — severity distribution donut chart, differential diagnosis confidence bar chart, agent timing bar chart — all populated from structured model output
- **Report panel** — severity banner, safety flags, all six report sections, finding cards color-coded by severity
- **DICOM metadata card** — study date, institution, modality, body part, technical parameters
- **PDF export** — full formatted report generated client-side
- **Clinical chat** — slide-up Q&A panel backed by the ClinicalAdvisorAgent
- **AMD GPU panel** — live util %, VRAM used/total, temperature, power draw — polling every 3 seconds

---

## Built For

**AMD Developer Hackathon 2026**
Track: Vision & Multimodal AI

This project demonstrates that AMD's ROCm ecosystem is a complete, production-viable alternative for serious AI workloads. Medical imaging analysis — with real multimodal vision, structured clinical reasoning, and standards-compliant output — running fully on AMD MI300X without a single NVIDIA or cloud dependency.

---

<div align="center">

**Built by Ramyar · Sulaymaniyah, Iraq**

*#AMDDevChallenge · AMD Instinct MI300X · ROCm · vLLM · Qwen*

</div>