Update README.md

README.md

@@ -1,3 +1,130 @@
 ---
 license: mit
+tags:
+- ophthalmology
+- object-detection
+- image-classification
+- medical-imaging
+- fundus
+- oct
+- detectron2
+- figure-parsing
+- pytorch
 ---
+
+# PubMed-Ophtha Detection Models
+
+This repository contains the three detection and classification models used in the
+[PubMed-Ophtha](https://arxiv.org/abs/2605.02720) dataset pipeline for parsing
+ophthalmological figures from scientific publications.
+
+**Paper:** Hallitschke V.J., Eickhoff C., Berens P. *PubMed-Ophtha: An open resource
+for training ophthalmology vision-language models on scientific literature.* arXiv:2605.02720 (2026).
+
+## Models
+
+The repository contains three model checkpoints under `models/`:
+
+| Directory | Checkpoint | Framework | Architecture | Task | Classes |
+|---|---|---|---|---|---|
+| `imaging_type_detection_1515892632/` | `model_0003909.pth` | PyTorch (Detectron2) | RetinaNet + ResNet FPN | Image type detection | CFP, OCT, Retinal Imaging, Other |
+| `panel_detection_1020880423/` | `model_0026865.pth` | PyTorch (Detectron2) | RetinaNet + ResNet FPN | Panel & identifier detection | Panel, Label |
+| `mark_status_classifier_482239176/` | `model_epoch_7.pth` | PyTorch | ResNet-50 | Mark status classification | Plain, Annotated |
+
+Each Detectron2 model directory also contains a `config.yaml` required for inference.
+
+### Panel Detection Model
+
+Detects panels and panel identifier labels (e.g. "A", "B") within multi-panel figures.
+Trained on the PubMed-Ophtha-Annotation dataset merged with
+[PanelSeg](https://doi.org/10.1145/3331184.3331253) and
+[ImageCLEF2016](https://www.imageclef.org/2016/medical), starting from an
+ImageCLEF2016-pretrained checkpoint.
+
+- **mAP@0.50:** 0.909 (panels), 0.903 (panel identifiers)
+- **mAP@0.95:** 0.532 (panels), 0.018 (panel identifiers)
+
+### Image Type Detection Model
+
+Detects individual images within a panel and assigns each a retinal imaging modality:
+color fundus photography (CFP), optical coherence tomography (OCT), retinal imaging
+(ultra-wide field / fluorescein angiography), or other (graphs, ultrasound, etc.).
+
+- **mAP@0.50:** 0.892
+- **mAP@0.95:** 0.558
+
+### Mark Status Classifier
+
+A ResNet-50 binary classifier applied to cropped image regions detected by the image
+type model. Predicts whether an image contains annotation marks such as arrows, dots,
+or bounding boxes.
+
+- **Accuracy:** 89.5% on the held-out test set
+
+## Usage
+
+Models are consumed by the [`pubmed-ophtha`](https://github.com/berenslab/pubmed-ophtha)
+Python package. Download all weights with:
+
+```bash
+pip install pubmed-ophtha
+pubmed-ophtha-split pull-models --local-dir .
+```
+
+Or download directly via `huggingface_hub`:
+
+```python
+from huggingface_hub import snapshot_download
+
+snapshot_download(repo_id="pubmed-ophtha/detection-models", local_dir=".")
+```
+
+After downloading, run inference via the `DetectronFigureSplitter`:
+
+```python
+from pubmed_ophtha.figure_splitting.detectron_figure_splitter import DetectronFigureSplitter
+from pubmed_ophtha.const.models import get_default_model_args
+
+splitter = DetectronFigureSplitter(**get_default_model_args())
+
+with open("figure.png", "rb") as f:
+    image_bytes = f.read()
+
+predictions = splitter.predict(image_bytes)
+# Keys: pred_boxes, pred_classes, scores,
+#       secondary_pred_classes, secondary_scores, keep_after_nms
+```
+
+`pred_classes` contains Panel/Label detections from the panel detection model followed
+by CFP/OCT/Retinal Imaging/Other detections from the image type model.
+`secondary_pred_classes` contains the Plain/Annotated mark status for each image
+detection (set to `"None"` for panel detections).
+
+## Training
+
+Both RetinaNet models use a ResNet backbone with FPN, finetuned from an
+ImageCLEF2016-pretrained Detectron2 checkpoint on the PubMed-Ophtha-Annotation
+dataset. The ResNet-50 classifier was trained from an ImageNet-pretrained checkpoint
+for 35 epochs with random cropping, flips, affine transformations, and color
+augmentation.
+
+## Dataset
+
+The ground-truth annotations used for training are available as part of the
+PubMed-Ophtha dataset:
+[huggingface.co/datasets/pubmed-ophtha/PubMed-Ophtha](https://huggingface.co/datasets/pubmed-ophtha/PubMed-Ophtha)
+
+## Citation
+
+```bibtex
+@article{hallitschke2026pubmed,
+  title={{PubMed-Ophtha}: An open resource for training ophthalmology vision-language models on scientific literature},
+  author={Hallitschke, Verena Jasmin and Eickhoff, Carsten and Berens, Philipp},
+  journal={arXiv preprint arXiv:2605.02720},
+  year={2026}
+}
+```
+
+## License
+
+MIT — see [LICENSE](LICENSE).