Add model card

README.md

@@ -0,0 +1,106 @@
+---
+license: apache-2.0
+language:
+- en
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- medical
+- clinical
+- healthcare
+- meditron
+- fully-open
+- medical-llm
+base_model: allenai/OLMo-2-0325-32B-SFT
+datasets:
+- EPFLiGHT/fully-open-meditron
+---
+
+# OLMo-2-32B-MeditronFO
+
+**OLMo-2-32B-MeditronFO** is a 32B-parameter medical specialist LLM, produced by supervised fine-tuning of [OLMo-2-32B-SFT](https://huggingface.co/allenai/OLMo-2-0325-32B-SFT) on the [Fully Open Meditron Corpus](https://huggingface.co/datasets/EPFLiGHT/fully-open-meditron).
+
+This model is part of the **Fully Open Meditron** family — the first end-to-end auditable pipeline for clinical LLMs, with open weights, open data, open training recipe, and clinician-vetted corpus construction.
+
+> OLMo-2-32B-MeditronFO improves +5.26 points over its base on aggregate medical benchmarks while preserving general-purpose capability.
+
+- 📄 **Paper:** *Fully Open Meditron: An Auditable Pipeline for Clinical LLMs*
+- 💻 **Code:** [github.com/EPFLiGHT/FullyOpenMeditron](https://github.com/EPFLiGHT/FullyOpenMeditron)
+- 📚 **Collection:** [Fully Open Meditron on the Hub](https://huggingface.co/EPFLiGHT)
+- 🗂️ **Training corpus:** [EPFLiGHT/fully-open-meditron](https://huggingface.co/datasets/EPFLiGHT/fully-open-meditron)
+
+## Performance
+
+Accuracy (%) on standard medical benchmarks. See the paper for full evaluation details, confidence intervals, and open-ended Auto-MOOVE results.
+
+| Benchmark | OLMo-2-32B-SFT | **OLMo-2-32B-MeditronFO** | Δ |
+|---|---:|---:|---:|
+| MedMCQA | 59.10 | **57.83** | -1.27 |
+| MedQA | 66.22 | **69.44** | +3.22 |
+| PubMedQA | 72.00 | **76.60** | +4.60 |
+| MedXpertQA | 13.02 | **17.96** | +4.94 |
+| HealthBench Hard | 19.75 | **33.82** | +14.07 |
+| **Average** | 45.88 | **51.13** | +5.25 |
+
+## Usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+
+model_id = "EPFLiGHT/OLMo-2-32B-MeditronFO"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+)
+
+messages = [
+    {"role": "user", "content": "A 62-year-old woman presents with a three-day history of dyspnea on exertion and a productive cough. What is the differential diagnosis?"},
+]
+prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+
+outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False)
+print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
+```
+
+## Training
+
+- **Base model:** [OLMo-2-32B-SFT](https://huggingface.co/allenai/OLMo-2-0325-32B-SFT)
+- **Corpus:** [Fully Open Meditron](https://huggingface.co/datasets/EPFLiGHT/fully-open-meditron) — ~601k examples (~150M tokens), aggregating eight public medical QA datasets with three clinician-vetted synthetic components: exam-style QA, guideline-grounded QA from 46,469 clinical practice guidelines, and open-ended clinical vignettes
+- **Hardware:** NVIDIA GH200 nodes
+- **Framework:** Axolotl with FSDP v2 / DeepSpeed ZeRO-3, Flash Attention 2, bf16 mixed precision
+- **Decontamination:** System-wide two-stage n-gram and token-alignment decontamination against all evaluation benchmarks
+
+Full hyperparameters are in Appendix I of the paper.
+
+## Intended Use
+
+**Research only.** This model is intended to support research on medical LLMs, auditing of clinical AI systems, and reproducibility of the Fully Open Meditron pipeline.
+
+It is **not validated for clinical deployment, individual patient advice, autonomous decision-making, or any other deployment-adjacent use.** Conduct independent domain-specific safety evaluation before any such use.
+
+## Limitations
+
+- Inherits limitations of the training corpus: ~64% of items are synthetic, generated by a single teacher model (gpt-oss-120b), introducing model-specific stylistic and reasoning biases.
+- Decontamination is syntactic (n-gram and token alignment) rather than semantic, leaving open the possibility of paraphrased leakage.
+- Some general-purpose instruction-following capability is degraded relative to the base model — a tradeoff common to medical-specialist fine-tuning.
+- Predominantly English; non-English clinical content is underrepresented.
+- Inherits geographic and demographic biases of the source datasets, partially mitigated by AfriMed-QA inclusion.
+- May produce confidently incorrect outputs (hallucinations).
+
+## Citation
+
+```bibtex
+@inproceedings{theimerlienhard2026meditron,
+  title={Fully Open Meditron: An Auditable Pipeline for Clinical LLMs},
+  author={Theimer-Lienhard, Xavier and El-Amin, Mushtaha and Elhassan, Fay and Vaidya, Sahaj and Cartier-Negadi, Victor and Sasu, David and Klein, Lars and Hartley, Mary-Anne},
+  year={2026}
+}
+```
+
+## License
+
+Released under the **apache-2.0** license. Permissive use including commercial, subject to attribution.