Image-to-Text
PEFT
Safetensors
medical-imaging
chest-xray
dermoscopy
vision-language
fairness
lora
mimic-cxr
padchest
ham10000
Instructions to use mbhosale/FairLLaVA with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use mbhosale/FairLLaVA with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
Add MIMIC-CXR subdir README with citations
Browse files- mimic-cxr/README.md +89 -9
mimic-cxr/README.md
CHANGED
|
@@ -2,25 +2,105 @@
|
|
| 2 |
license: apache-2.0
|
| 3 |
library_name: peft
|
| 4 |
base_model: lmsys/vicuna-7b-v1.5
|
|
|
|
| 5 |
tags:
|
| 6 |
- medical-imaging
|
| 7 |
- chest-xray
|
| 8 |
- mimic-cxr
|
|
|
|
| 9 |
- fairness
|
| 10 |
- lora
|
|
|
|
|
|
|
|
|
|
| 11 |
---
|
| 12 |
|
| 13 |
# FairLLaVA — MIMIC-CXR
|
| 14 |
|
| 15 |
Fairness-aware LoRA adapter on top of LLaVA-Rad (Vicuna-7B + BiomedCLIP-CXR-518)
|
| 16 |
-
for MIMIC-CXR chest-X-ray report generation. Trained with the FairLLaVA
|
| 17 |
-
mutual-information regularizer on patient demographics (age, sex, race)
|
|
|
|
| 18 |
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
- `mm_projector.bin` — multimodal projector
|
| 22 |
-
- `tokenizer.model`, `tokenizer_config.json`, `special_tokens_map.json`, `config.json` — tokenizer / config
|
| 23 |
|
| 24 |
-
##
|
| 25 |
-
|
| 26 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2 |
license: apache-2.0
|
| 3 |
library_name: peft
|
| 4 |
base_model: lmsys/vicuna-7b-v1.5
|
| 5 |
+
pipeline_tag: image-to-text
|
| 6 |
tags:
|
| 7 |
- medical-imaging
|
| 8 |
- chest-xray
|
| 9 |
- mimic-cxr
|
| 10 |
+
- vision-language
|
| 11 |
- fairness
|
| 12 |
- lora
|
| 13 |
+
- peft
|
| 14 |
+
datasets:
|
| 15 |
+
- physionet/mimic-cxr-jpg
|
| 16 |
---
|
| 17 |
|
| 18 |
# FairLLaVA — MIMIC-CXR
|
| 19 |
|
| 20 |
Fairness-aware LoRA adapter on top of LLaVA-Rad (Vicuna-7B + BiomedCLIP-CXR-518)
|
| 21 |
+
for **MIMIC-CXR** chest-X-ray report generation. Trained with the FairLLaVA
|
| 22 |
+
mutual-information regularizer on patient demographics (age, sex, race) to
|
| 23 |
+
reduce inter-group performance gaps while preserving clinical accuracy.
|
| 24 |
|
| 25 |
+
Code: [github.com/bhosalems/FairLLaVA](https://github.com/bhosalems/FairLLaVA)
|
| 26 |
+
Paper: [arxiv.org/abs/2603.26008](https://arxiv.org/abs/2603.26008)
|
|
|
|
|
|
|
| 27 |
|
| 28 |
+
## Files in this directory
|
| 29 |
+
|
| 30 |
+
| File | Purpose |
|
| 31 |
+
|---|---|
|
| 32 |
+
| `adapter_model.safetensors`, `adapter_config.json` | LoRA adapter weights + config |
|
| 33 |
+
| `non_lora_trainables.bin` | non-LoRA trainable params (projector + token embeddings) |
|
| 34 |
+
| `mm_projector.bin` | multimodal projector (vision -> LLM token space) |
|
| 35 |
+
| `config.json` | LLaVA model config |
|
| 36 |
+
| `tokenizer.model`, `tokenizer_config.json`, `special_tokens_map.json` | Vicuna tokenizer |
|
| 37 |
+
|
| 38 |
+
## Quick start
|
| 39 |
+
|
| 40 |
+
```python
|
| 41 |
+
from huggingface_hub import snapshot_download
|
| 42 |
+
from llava.model.builder import load_pretrained_model
|
| 43 |
+
|
| 44 |
+
local_dir = snapshot_download(
|
| 45 |
+
repo_id="mbhosale/FairLLaVA",
|
| 46 |
+
allow_patterns="mimic-cxr/*",
|
| 47 |
+
)
|
| 48 |
+
|
| 49 |
+
tokenizer, model, image_processor, ctx_len = load_pretrained_model(
|
| 50 |
+
f"{local_dir}/mimic-cxr",
|
| 51 |
+
model_base="lmsys/vicuna-7b-v1.5",
|
| 52 |
+
model_name="llavarad",
|
| 53 |
+
)
|
| 54 |
+
```
|
| 55 |
+
|
| 56 |
+
See the full inference example in [`inference.py`](https://github.com/bhosalems/FairLLaVA/blob/main/inference.py).
|
| 57 |
+
|
| 58 |
+
## Ethics
|
| 59 |
+
|
| 60 |
+
This checkpoint is released **for research and educational use only**. It is
|
| 61 |
+
**not** approved or validated for clinical or diagnostic use and must not be
|
| 62 |
+
used to make medical decisions or to inform patient care. Use of MIMIC-CXR is
|
| 63 |
+
governed by the PhysioNet data-use agreement.
|
| 64 |
+
|
| 65 |
+
## Citation
|
| 66 |
+
|
| 67 |
+
If you use this checkpoint, please cite FairLLaVA and the upstream works it builds on:
|
| 68 |
+
|
| 69 |
+
```bibtex
|
| 70 |
+
@misc{bhosale2026fairllava,
|
| 71 |
+
title={FairLLaVA: Fairness-Aware Parameter-Efficient Fine-Tuning for Large Vision-Language Assistants},
|
| 72 |
+
author={Mahesh Bhosale and Abdul Wasi and Shantam Srivastava and Shifa Latif and Tianyu Luan and Mingchen Gao and David Doermann and Xuan Gong},
|
| 73 |
+
year={2026},
|
| 74 |
+
eprint={2603.26008},
|
| 75 |
+
archivePrefix={arXiv},
|
| 76 |
+
primaryClass={cs.CV},
|
| 77 |
+
url={https://arxiv.org/abs/2603.26008}
|
| 78 |
+
}
|
| 79 |
+
|
| 80 |
+
@article{ZambranoChaves2025,
|
| 81 |
+
title={A clinically accessible small multimodal radiology model and evaluation metric for chest X-ray findings},
|
| 82 |
+
author={Zambrano Chaves, Juan Manuel and others},
|
| 83 |
+
journal={Nature Communications}, year={2025}, volume={16}, pages={3108},
|
| 84 |
+
doi={10.1038/s41467-025-58344-x}
|
| 85 |
+
}
|
| 86 |
+
|
| 87 |
+
@misc{liu2023improvedllava,
|
| 88 |
+
title={Improved Baselines with Visual Instruction Tuning},
|
| 89 |
+
author={Liu, Haotian and Li, Chunyuan and Li, Yuheng and Lee, Yong Jae},
|
| 90 |
+
publisher={arXiv:2310.03744},
|
| 91 |
+
year={2023}
|
| 92 |
+
}
|
| 93 |
+
|
| 94 |
+
@article{johnson2019mimic,
|
| 95 |
+
title={MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports},
|
| 96 |
+
author={Johnson, Alistair E. W. and Pollard, Tom J. and Berkowitz, Seth J. and Greenbaum, Nathaniel R. and Lungren, Matthew P. and Deng, Chih-ying and Mark, Roger G. and Horng, Steven},
|
| 97 |
+
journal={Scientific Data}, year={2019}, volume={6}, number={1}, pages={317},
|
| 98 |
+
doi={10.1038/s41597-019-0322-0}
|
| 99 |
+
}
|
| 100 |
+
|
| 101 |
+
@article{johnson2019mimiccxrjpg,
|
| 102 |
+
title={MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs},
|
| 103 |
+
author={Johnson, Alistair E. W. and Pollard, Tom J. and Greenbaum, Nathaniel R. and Lungren, Matthew P. and Deng, Chih-ying and Peng, Yifan and Lu, Zhiyong and Mark, Roger G. and Berkowitz, Seth J. and Horng, Steven},
|
| 104 |
+
journal={arXiv preprint arXiv:1901.07042}, year={2019}
|
| 105 |
+
}
|
| 106 |
+
```
|