File size: 2,506 Bytes
cb5dc6b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 | ---
license: cc-by-nc-4.0
tags:
- diffusion
- inpainting
- multimodal
- autonomous-driving
- nuscenes
---
# 🐳 MObI: Multimodal Object Inpainting Using Diffusion Models
Pretrained weights for **MObI**, a diffusion-based model for joint multimodal object inpainting across camera and lidar, conditioned on a single reference image and a 3D bounding box.
📄 **Paper:** [arXiv:2501.03173](https://arxiv.org/abs/2501.03173)
💻 **Code:** [github.com/alexbuburuzan/MObI](https://github.com/alexbuburuzan/MObI)
**Venue:** CVPR Workshop on Data-Driven Autonomous Driving Simulation (DDADS), 2025
## Overview
MObI extends [Paint-by-Example](https://github.com/Fantasy-Studio/Paint-by-Example) to:
- Jointly inpaint **RGB camera, lidar depth, and lidar intensity**
- Insert objects from a **single reference image**
- Use **3D bounding box conditioning** for accurate spatial placement
This combines the realism of reference-based inpainting with the controllability of 3D-aware methods.
## Contents
| File | Description |
|------|-------------|
| `mobi_nuscenes_epoch28.ckpt` | MObI trained on nuScenes |
| `autoencoders/range_autoencoder.ckpt` | Range-view VAE for lidar |
## Results (nuScenes)
| Reference Type | FID ↓ | LPIPS ↓ | CLIP ↑ | D-LPIPS ↓ | I-LPIPS ↓ |
|---------------|-------|---------|--------|-----------|-----------|
| id-ref | 6.503 | 0.114 | 84.9 | 0.130 | 0.147 |
| track-ref | 6.703 | 0.115 | 83.5 | 0.129 | 0.149 |
| in-domain-ref | 8.947 | 0.127 | 77.5 | 0.132 | 0.154 |
| cross-domain-ref | 9.046 | 0.130 | 76.0 | 0.132 | 0.153 |
## Usage
See the [GitHub repository](https://github.com/alexbuburuzan/MObI) for installation, data preprocessing, inference, and training instructions.
```bash
git clone https://github.com/alexbuburuzan/MObI.git
cd MObI
bash scripts/download_models.sh
bash scripts/realism_test_bench.sh
```
## Citation
```bibtex
@InProceedings{Buburuzan_2025_CVPR,
author = {Buburuzan, Alexandru and Sharma, Anuj and Redford, John and Dokania, Puneet K. and Mueller, Romain},
title = {MObI: Multimodal Object Inpainting Using Diffusion Models},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
month = {June},
year = {2025},
pages = {1999-2009}
}
```
## License
Released under [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/). Note that this work builds on Paint-by-Example and BEVFusion, which have their own licenses. |