| --- |
| license: mit |
| tags: |
| - ophthalmology |
| - object-detection |
| - image-classification |
| - medical-imaging |
| - fundus |
| - oct |
| - detectron2 |
| - figure-parsing |
| - pytorch |
| --- |
| |
| # PubMed-Ophtha Detection Models |
|
|
| This repository contains the three detection and classification models used in the |
| [PubMed-Ophtha](https://arxiv.org/abs/2605.02720) dataset pipeline for parsing |
| ophthalmological figures from scientific publications. |
|
|
| **Paper:** Hallitschke V.J., Eickhoff C., Berens P. *PubMed-Ophtha: An open resource |
| for training ophthalmology vision-language models on scientific literature.* arXiv:2605.02720 (2026). |
|
|
| ## Models |
|
|
| The repository contains three model checkpoints under `models/`: |
|
|
| | Directory | Checkpoint | Framework | Architecture | Task | Classes | |
| |---|---|---|---|---|---| |
| | `imaging_type_detection_1515892632/` | `model_0003909.pth` | PyTorch (Detectron2) | RetinaNet + ResNet FPN | Image type detection | CFP, OCT, Retinal Imaging, Other | |
| | `panel_detection_1020880423/` | `model_0026865.pth` | PyTorch (Detectron2) | RetinaNet + ResNet FPN | Panel & identifier detection | Panel, Label | |
| | `mark_status_classifier_482239176/` | `model_epoch_7.pth` | PyTorch | ResNet-50 | Mark status classification | Plain, Annotated | |
|
|
| Each Detectron2 model directory also contains a `config.yaml` required for inference. |
|
|
| ### Panel Detection Model |
|
|
| Detects panels and panel identifier labels (e.g. "A", "B") within multi-panel figures. |
| Trained on the PubMed-Ophtha-Annotation dataset merged with |
| [PanelSeg](https://doi.org/10.1145/3331184.3331253) and |
| [ImageCLEF2016](https://www.imageclef.org/2016/medical), starting from an |
| ImageCLEF2016-pretrained checkpoint. |
|
|
| - **mAP@0.50:** 0.909 (panels), 0.903 (panel identifiers) |
| - **mAP@0.95:** 0.532 (panels), 0.018 (panel identifiers) |
|
|
| ### Image Type Detection Model |
|
|
| Detects individual images within a panel and assigns each a retinal imaging modality: |
| color fundus photography (CFP), optical coherence tomography (OCT), retinal imaging |
| (ultra-wide field / fluorescein angiography), or other (graphs, ultrasound, etc.). |
|
|
| - **mAP@0.50:** 0.892 |
| - **mAP@0.95:** 0.558 |
|
|
| ### Mark Status Classifier |
|
|
| A ResNet-50 binary classifier applied to cropped image regions detected by the image |
| type model. Predicts whether an image contains annotation marks such as arrows, dots, |
| or bounding boxes. |
|
|
| - **Accuracy:** 89.5% on the held-out test set |
|
|
| ## Usage |
|
|
| Models are consumed by the [`pubmed-ophtha`](https://github.com/berenslab/pubmed-ophtha) |
| Python package. Download all weights with: |
|
|
| ```bash |
| pip install pubmed-ophtha |
| pubmed-ophtha-split pull-models --local-dir . |
| ``` |
|
|
| Or download directly via `huggingface_hub`: |
|
|
| ```python |
| from huggingface_hub import snapshot_download |
| |
| snapshot_download(repo_id="pubmed-ophtha/detection-models", local_dir=".") |
| ``` |
|
|
| After downloading, run inference via the `DetectronFigureSplitter`: |
|
|
| ```python |
| from pubmed_ophtha.figure_splitting.detectron_figure_splitter import DetectronFigureSplitter |
| from pubmed_ophtha.const.models import get_default_model_args |
| |
| splitter = DetectronFigureSplitter(**get_default_model_args()) |
| |
| with open("figure.png", "rb") as f: |
| image_bytes = f.read() |
| |
| predictions = splitter.predict(image_bytes) |
| # Keys: pred_boxes, pred_classes, scores, |
| # secondary_pred_classes, secondary_scores, keep_after_nms |
| ``` |
|
|
| `pred_classes` contains Panel/Label detections from the panel detection model followed |
| by CFP/OCT/Retinal Imaging/Other detections from the image type model. |
| `secondary_pred_classes` contains the Plain/Annotated mark status for each image |
| detection (set to `"None"` for panel detections). |
|
|
| ## Training |
|
|
| Both RetinaNet models use a ResNet backbone with FPN, finetuned from an |
| ImageCLEF2016-pretrained Detectron2 checkpoint on the PubMed-Ophtha-Annotation |
| dataset. The ResNet-50 classifier was trained from an ImageNet-pretrained checkpoint |
| for 35 epochs with random cropping, flips, affine transformations, and color |
| augmentation. |
|
|
| ## Dataset |
|
|
| The ground-truth annotations used for training are available as part of the |
| PubMed-Ophtha dataset: |
| [huggingface.co/datasets/pubmed-ophtha/PubMed-Ophtha](https://huggingface.co/datasets/pubmed-ophtha/PubMed-Ophtha) |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{hallitschke2026pubmed, |
| title={{PubMed-Ophtha}: An open resource for training ophthalmology vision-language models on scientific literature}, |
| author={Hallitschke, Verena Jasmin and Eickhoff, Carsten and Berens, Philipp}, |
| journal={arXiv preprint arXiv:2605.02720}, |
| year={2026} |
| } |
| ``` |
|
|
| ## License |
|
|
| MIT — see [LICENSE](LICENSE). |
