| --- |
| license: mit |
| task_categories: |
| - visual-question-answering |
| language: |
| - en |
| tags: |
| - hallucination-detection |
| - object-hallucination |
| - pope |
| - coco |
| - benchmark |
| size_categories: |
| - 1K<n<10K |
| --- |
| |
| # RePOPE: Revisiting Partial Object Hallucination Evaluation |
|
|
| RePOPE is a re-annotated version of the POPE (Polling-based Object Probing Evaluation) benchmark with corrected ground-truth labels. It evaluates object hallucination in multimodal large language models (MLLMs) by asking yes/no questions about object existence in MSCOCO images. |
|
|
| ## Dataset Details |
|
|
| - **Original Paper:** [RePOPE: Revisiting Partial Object Hallucination Evaluation](https://arxiv.org/abs/2405.14571) |
| - **Original Repository:** [https://github.com/YanNeu/RePOPE](https://github.com/YanNeu/RePOPE) |
| - **Images:** MSCOCO 2014 (subset of 500 images) |
|
|
| ## Dataset Structure |
|
|
| Each row contains: |
|
|
| - `image`: The MSCOCO image (struct with `bytes` and `path`) |
| - `image_id`: COCO image identifier (e.g., `000000310196`) |
| - `question`: A yes/no question about object presence (e.g., "Is there a snowboard in the image?") |
| - `answer`: Ground truth label (`yes` or `no`) |
| - `category`: Sampling strategy used to select the queried object (`random`, `popular`, or `adversarial`) |
|
|
| ### Splits |
|
|
| This dataset contains all three POPE sampling categories in a single split: |
|
|
| | Category | Count | |
| |---------------|-------| |
| | random | 2,774 | |
| | popular | 2,727 | |
| | adversarial | 2,684 | |
| | **Total** | **8,185** | |
|
|
| ### Label Distribution |
|
|
| | Answer | Count | |
| |--------|-------| |
| | yes | 3,539 | |
| | no | 4,646 | |
|
|
| ## How to Use |
|
|
| ```python |
| from datasets import load_dataset |
| |
| ds = load_dataset("MM-Hallu/RePOPE") |
| ``` |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{neuhaus2024repope, |
| title={RePOPE: Revisiting Partial Object Hallucination Evaluation}, |
| author={Yannik Neuschwander and Selen Yu and Jordy Van Landeghem and Jan Van Loock and Lilian Ngweta and Rukiye Savran Kizildag and Desmond Elliott and Matthew B. Blaschko}, |
| year={2024}, |
| eprint={2405.14571}, |
| archivePrefix={arXiv}, |
| primaryClass={cs.CV} |
| } |
| ``` |
|
|
