RadTriage AI — MedGemma 4B for Radiology Triage & Report Drafting
A prompt-engineered pipeline built on top of MedGemma 4B-IT from Google's Health AI Developer Foundations (HAI-DEF) for multimodal radiology triage and report generation.
Model Description
RadTriage AI uses MedGemma 4B-IT in a two-pass architecture designed to reduce hallucination in medical image interpretation:
- Pass 1 (Vision+Text): Extracts structured findings as JSON with anatomical location, severity, and calibrated confidence scores
- Pass 2 (Text-only): Generates a narrative radiology report grounded in Pass 1 findings — cannot hallucinate findings not detected in Pass 1
- Triage: Classifies urgency as CRITICAL / URGENT / ROUTINE with rationale
Base Model
This model traces to google/medgemma-4b-it — a multimodal medical AI model from Google's HAI-DEF collection, specifically designed for medical image interpretation.
How to Use
import torch
import json
from PIL import Image
from transformers import AutoProcessor, AutoModelForImageTextToText
# Load the base MedGemma model
model_id = "google/medgemma-4b-it"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(model_id)
# Load the RadTriage prompt configuration
# (See radtriage_config.json for full prompt templates)
image = Image.open("chest_xray.png").convert("RGB")
# Pass 1: Structured Findings
pass1_prompt = """You are an expert radiologist assistant analyzing a Chest X-ray image.
Clinical indication: Shortness of breath
Prior studies: None
Specifically evaluate for: pneumothorax, pleural effusion, consolidation,
cardiomegaly, mediastinal widening, rib fractures, pulmonary edema, masses,
nodules, and lines/tubes.
Analyze the image systematically and respond ONLY with a valid JSON object
following this exact schema:
{
"modality_detected": "<specific modality>",
"anatomical_region": "<body region>",
"image_quality": "<adequate|suboptimal|non-diagnostic>",
"quality_issues": [],
"comparison_available": false,
"findings": [
{"id": 1, "description": "<finding>", "location": "<location>",
"severity": "<normal|mild|moderate|severe>", "confidence": 0.0}
],
"overall_impression_category": "<normal|abnormal|critical>",
"missing_context": []
}"""
messages = [{"role": "user", "content": [
{"type": "image", "image": image},
{"type": "text", "text": pass1_prompt}
]}]
inputs = processor.apply_chat_template(
messages, add_generation_prompt=True, tokenize=True,
return_dict=True, return_tensors="pt"
).to(model.device)
with torch.inference_mode():
output_ids = model.generate(**inputs, max_new_tokens=1024, do_sample=False)
input_len = inputs["input_ids"].shape[-1]
result = processor.decode(output_ids[0][input_len:], skip_special_tokens=True)
findings = json.loads(result)
# Pass 2: Narrative Report (text-only, no image)
pass2_prompt = f"""You are a radiologist drafting a formal report. Convert the
following structured findings into a professional radiology report following
ACR reporting guidelines.
STRUCTURED FINDINGS:
{json.dumps(findings, indent=2)}
Generate sections: EXAMINATION, CLINICAL INDICATION, COMPARISON, FINDINGS, IMPRESSION.
ONLY describe findings from the structured data. Do NOT invent new findings."""
messages_p2 = [{"role": "user", "content": [{"type": "text", "text": pass2_prompt}]}]
# ... generate report
Pipeline Configuration
The radtriage_config.json file contains:
- Modality-aware prompt templates for 5 imaging types
- Triage classification rules and critical keywords
- Confidence calibration thresholds
- Missing context checklist templates
Supported Modalities
| Modality | Specific Evaluations |
|---|---|
| Chest X-ray | Pneumothorax, effusion, consolidation, cardiomegaly, lines/tubes |
| CT Head | Hemorrhage, mass effect, midline shift, hydrocephalus, fractures |
| CT Chest | Nodules, ground glass, PE, lymphadenopathy |
| MRI Brain | DWI restriction, enhancement, FLAIR hyperintensity |
| Histopathology | Architecture, atypia, margins, grade |
Evaluation Results
Tested on 3 real NIH Chest X-ray cases:
| Case | Indication | Findings | Triage | Pipeline Time |
|---|---|---|---|---|
| 1 | SOB + cough | 3 (2 abnormal) | URGENT | 389s |
| 2 | Chest pain | 8 (0 abnormal) | ROUTINE | 627s |
| 3 | Pre-op clearance | 9 (2 abnormal) | ROUTINE | 783s |
Key Innovation: Two-Pass Anti-Hallucination
Traditional single-pass approaches generate reports directly from images, which can hallucinate findings. Our two-pass approach:
- Pass 1 generates structured JSON (auditable, machine-readable)
- Pass 2 generates prose ONLY from the JSON (cannot invent findings)
This architectural constraint is a safety feature for clinical AI applications.
Links
- Competition: MedGemma Impact Challenge
- Kaggle Notebook: RadTriage AI Pipeline
- GitHub: github.com/pramodmisra/radtriage-ai
- Live Demo: HuggingFace Spaces
Citation
@software{radtriage_ai_2026,
title={RadTriage AI: Multimodal Radiology Triage & Report Drafting},
author={Pramod Misra},
year={2026},
url={https://github.com/pramodmisra/radtriage-ai}
}
Disclaimer
⚠️ For demonstration and research purposes only. Not intended for clinical use, diagnosis, or treatment decisions. All medical image interpretations should be performed by qualified healthcare professionals.
Use of MedGemma is subject to the HAI-DEF Terms of Use.
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support