README.md

---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- medical
- clinical
- healthcare
- meditron
- fully-open
- medical-llm
base_model: swiss-ai/Apertus-70B-Instruct-2509
base_model_relation: finetune
datasets:
- EPFLiGHT/fully-open-meditron
---

# Apertus-70B-MeditronFO

**Apertus-70B-MeditronFO** is a 70B-parameter medical specialist LLM, produced by supervised fine-tuning of [Apertus-70B-Instruct](https://huggingface.co/swiss-ai/Apertus-70B-Instruct-2509) on the [Fully Open Meditron Corpus](https://huggingface.co/datasets/EPFLiGHT/fully-open-meditron).

This model is part of the **Fully Open Meditron** family — the first end-to-end auditable pipeline for clinical LLMs, with open weights, open data, open training recipe, and clinician-vetted corpus construction.

> Apertus-70B-MeditronFO establishes a new state of the art among fully open medical LLMs, and is preferred over Llama-3.1-70B-Meditron in 96.6% of pairwise Auto-MOOVE comparisons.

- 📄 **Paper:** [*Fully Open Meditron: An Auditable Pipeline for Clinical LLMs*](https://arxiv.org/abs/2605.16215)
- 💻 **Code:** [github.com/EPFLiGHT/FullyOpenMeditron](https://github.com/EPFLiGHT/FullyOpenMeditron)
- 📚 **Collection:** [MeditronFO](https://huggingface.co/collections/EPFLiGHT/meditronfo)
- 🗂️ **Training corpus:** [EPFLiGHT/fully-open-meditron](https://huggingface.co/datasets/EPFLiGHT/fully-open-meditron)

## Performance

Accuracy (%) on standard medical benchmarks. See the paper for full evaluation details, confidence intervals, and open-ended Auto-MOOVE results.

| Benchmark | Apertus-70B-Instruct | **Apertus-70B-MeditronFO** | Δ |
|---|---:|---:|---:|
| MedMCQA | 52.43 | **56.32** | +3.89 |
| MedQA | 60.64 | **68.58** | +7.94 |
| PubMedQA | 66.80 | **75.20** | +8.40 |
| MedXpertQA | 12.33 | **16.90** | +4.57 |
| HealthBench Hard | 32.28 | **40.14** | +7.86 |
| **Average** | 44.90 | **51.43** | +6.53 |

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "EPFLiGHT/Apertus-70B-MeditronFO"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [
    {"role": "user", "content": "A 62-year-old woman presents with a three-day history of dyspnea on exertion and a productive cough. What is the differential diagnosis?"},
]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
```

## Training

- **Base model:** [Apertus-70B-Instruct](https://huggingface.co/swiss-ai/Apertus-70B-Instruct-2509)
- **Corpus:** [Fully Open Meditron](https://huggingface.co/datasets/EPFLiGHT/fully-open-meditron) — ~601k examples (~150M tokens), aggregating eight public medical QA datasets with three clinician-vetted synthetic components: exam-style QA, guideline-grounded QA from 46,469 clinical practice guidelines, and open-ended clinical vignettes
- **Hardware:** NVIDIA GH200 nodes
- **Framework:** Axolotl with FSDP v2 / DeepSpeed ZeRO-3, Flash Attention 2, bf16 mixed precision
- **Decontamination:** System-wide two-stage n-gram and token-alignment decontamination against all evaluation benchmarks

Full hyperparameters are in Appendix I of the paper.

## Intended Use

**Research only.** This model is intended to support research on medical LLMs, auditing of clinical AI systems, and reproducibility of the Fully Open Meditron pipeline.

It is **not validated for clinical deployment, individual patient advice, autonomous decision-making, or any other deployment-adjacent use.** Conduct independent domain-specific safety evaluation before any such use.

## Citation

If you use this model, please cite:

```bibtex
@misc{theimerlienhard2026fullyopenmeditronauditable,
  title         = {Fully Open Meditron: An Auditable Pipeline for Clinical LLMs},
  author        = {Xavier Theimer-Lienhard and Mushtaha El-Amin and Fay Elhassan and Sahaj Vaidya and Victor Cartier-Negadi and David Sasu and Lars Klein and Mary-Anne Hartley},
  year          = {2026},
  eprint        = {2605.16215},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url           = {https://arxiv.org/abs/2605.16215}
}
```

## License

Released under the **apache-2.0** license. Permissive use including commercial, subject to attribution.