File size: 2,110 Bytes
56d2c5a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 | ---
license: mit
task_categories:
- visual-question-answering
language:
- en
tags:
- hallucination-detection
- object-hallucination
- pope
- coco
- benchmark
size_categories:
- 1K<n<10K
---
# RePOPE: Revisiting Partial Object Hallucination Evaluation
RePOPE is a re-annotated version of the POPE (Polling-based Object Probing Evaluation) benchmark with corrected ground-truth labels. It evaluates object hallucination in multimodal large language models (MLLMs) by asking yes/no questions about object existence in MSCOCO images.
## Dataset Details
- **Original Paper:** [RePOPE: Revisiting Partial Object Hallucination Evaluation](https://arxiv.org/abs/2405.14571)
- **Original Repository:** [https://github.com/YanNeu/RePOPE](https://github.com/YanNeu/RePOPE)
- **Images:** MSCOCO 2014 (subset of 500 images)
## Dataset Structure
Each row contains:
- `image`: The MSCOCO image (struct with `bytes` and `path`)
- `image_id`: COCO image identifier (e.g., `000000310196`)
- `question`: A yes/no question about object presence (e.g., "Is there a snowboard in the image?")
- `answer`: Ground truth label (`yes` or `no`)
- `category`: Sampling strategy used to select the queried object (`random`, `popular`, or `adversarial`)
### Splits
This dataset contains all three POPE sampling categories in a single split:
| Category | Count |
|---------------|-------|
| random | 2,774 |
| popular | 2,727 |
| adversarial | 2,684 |
| **Total** | **8,185** |
### Label Distribution
| Answer | Count |
|--------|-------|
| yes | 3,539 |
| no | 4,646 |
## How to Use
```python
from datasets import load_dataset
ds = load_dataset("MM-Hallu/RePOPE")
```
## Citation
```bibtex
@misc{neuhaus2024repope,
title={RePOPE: Revisiting Partial Object Hallucination Evaluation},
author={Yannik Neuschwander and Selen Yu and Jordy Van Landeghem and Jan Van Loock and Lilian Ngweta and Rukiye Savran Kizildag and Desmond Elliott and Matthew B. Blaschko},
year={2024},
eprint={2405.14571},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
```
|