Update README.md

README.md

@@ -4,11 +4,119 @@ pipeline_tag: image-to-image
 library_name: diffusers
 ---
 
-# [ICCV 2025] GenStereo: Towards Open-World Generation of Stereo Images and Unsupervised Matching
-
+# [ICCV 2025] Towards Open-World Generation of Stereo Images and Unsupervised Matching
 
 [![Project Site](https://img.shields.io/badge/Project-Web-green)](https://qjizhi.github.io/genstereo)  [![Spaces](https://img.shields.io/badge/Spaces-Demo-yellow?logo=huggingface)](https://huggingface.co/spaces/FQiao/GenStereo)  [![Github](https://img.shields.io/badge/Github-Repo-orange?logo=github)](https://github.com/Qjizhi/GenStereo)  [![Models](https://img.shields.io/badge/Models-checkpoints-blue?logo=huggingface)](https://huggingface.co/FQiao/GenStereo-sd2.1/tree/main)  [![arXiv](https://img.shields.io/badge/arXiv-2503.12720-red?logo=arxiv)](https://arxiv.org/abs/2503.12720)
 
+This repository contains the model presented in [Towards Open-World Generation of Stereo Images and Unsupervised Matching](https://huggingface.co/papers/2503.12720). The models are finetuned on Stable Diffusion 1.5, for SD v2.1, you can find [here](https://huggingface.co/FQiao/GenStereo-sd2.1).
+![](teaser_coco.jpg)
+
+## Abstract
+Stereo images are fundamental to numerous applications, including extended reality (XR) devices, autonomous driving, and robotics. Unfortunately, acquiring high-quality stereo images remains challenging due to the precise calibration requirements of dual-camera setups and the complexity of obtaining accurate, dense disparity maps. Existing stereo image generation methods typically focus on either visual quality for viewing or geometric accuracy for matching, but not both. We introduce GenStereo, a diffusion-based approach, to bridge this gap. The method includes two primary innovations (1) conditioning the diffusion process on a disparity-aware coordinate embedding and a warped input image, allowing for more precise stereo alignment than previous methods, and (2) an adaptive fusion mechanism that intelligently combines the diffusion-generated image with a warped image, improving both realism and disparity consistency. Through extensive training on 11 diverse stereo datasets, GenStereo demonstrates strong generalization ability. GenStereo achieves state-of-the-art performance in both stereo image generation and unsupervised stereo matching tasks.
+
+## How to use
+
+### Environment
+
+We tested our codes on Ubuntu with nVidia A100 GPU. If you're using other machines like Windows, consider using Docker. You can either add packages to your python environment or use Docker to build an python environment. Commands below are all expected to run in the root directory of the repository.
+
+We tested the environment with python `>=3.10` and CUDA `=11.8`. To add mandatory dependencies run the command below.
+
+``` shell
+pip install -r requirements.txt
+```
+
+To run developmental codes such as the example provided in jupyter notebook and the live demo implemented by gradio, add extra dependencies via the command below.
+
+``` shell
+pip install -r requirements_dev.txt
+```
+
+### Download pretrained models
+
+GenStereo uses pretrained models which consist of both our finetuned models and publicly available third-party ones. Download all the models to `checkpoints` directory or anywhere of your choice. You can do it manually or by the [download_models.sh](scripts/download_models.sh) script.
+
+#### Download script
+
+``` shell
+bash scripts/download_models.sh 
+```
+
+#### Manual download
+
+> [!NOTE]
+> Models and checkpoints provided below may be distributed under different licenses. Users are required to check licenses carefully on their behalf.
+
+1. Our finetuned models, we provide two versions of GenStereo
+    - v1.5: 512px, faster, [model card](https://huggingface.co/FQiao/GenStereo).
+    - v2.1: 768px, better performance, high resolution, takes more time, [model card](https://huggingface.co/FQiao/GenStereo-sd2.1).
+2. Pretrained models:
+    - [sd-vae-ft-mse](https://huggingface.co/stabilityai/sd-vae-ft-mse)
+      - download `config.json` and `diffusion_pytorch_model.safetensors` to `checkpoints/sd-vae-ft-mse`
+    - [sd-image-variations-diffusers](https://huggingface.co/lambdalabs/sd-image-variations-diffusers)
+      - download `image_encoder/config.json` and `image_encoder/pytorch_model.bin` to `checkpoints/image_encoder`
+3. MDE (Monocular Depth Estimation) models
+    - We use [Depth Anything V2](https://github.com/DepthAnything/Depth-Anything-V2) as the MDE model and get the disparity maps.
+The final `checkpoints` directory must look like this:
+
+```
+.
+├── depth_anything_v2_vitl.pth
+├── genstereo-v1.5
+│   ├── config.json
+│   ├── denoising_unet.pth
+│   ├── fusion_layer.pth
+│   ├── pose_guider.pth
+│   └── reference_unet.pth
+├── genstereo-v2.1
+│   ├── config.json
+│   ├── denoising_unet.pth
+│   ├── fusion_layer.pth
+│   ├── pose_guider.pth
+│   └── reference_unet.pth
+├── image_encoder
+│   ├── config.json
+│   └── pytorch_model.bin
+└── sd-vae-ft-mse
+    ├── config.json
+    └── diffusion_pytorch_model.safetensors
+```
+
+### Inference
+You can easily run the inference code by running the following command, and the results will be save under `./vis` folder.
+
+```bash
+python test.py /path/to/your/image
+```
+
+### Gradio live demo
+
+An interactive live demo is also available. Start gradio demo by running the command below, and goto [http://127.0.0.1:7860/](http://127.0.0.1:7860/)
+If you are running it on the server, be sure to forward the port 7860.
+
+Or you can just visit [Spaces](https://huggingface.co/spaces/FQiao/GenStereo) hosted by Hugging Face to try it now.
+
+```shell
+python app.py
+```
+
+## Train
+Please read [Train_Guide.md](./Trian_Guide.md).
+
+## Citation
+
+``` bibtex
+  @inproceedings{qiao2025genstereo,
+    author    = {Qiao, Feng and Xiong, Zhexiao and Xing, Eric and Jacobs, Nathan},
+    title     = {Towards Open-World Generation of Stereo Images and Unsupervised Matching},
+    booktitle = {Proceedings of the {IEEE/CVF} International Conference on Computer Vision ({ICCV})},
+    year      = {2025},
+    eprint    = {2503.12720},
+    archiveprefix = {arXiv},
+    primaryclass = {cs.CV}
+  }
+```
+
+## Acknowledgements
 
-This repository contains the model presented in [GenStereo: Towards Open-World Generation of Stereo Images and Unsupervised Matching](https://huggingface.co/papers/2503.12720). The models are finetuned on Stable Diffusion 1.5, for SD v2.1, you can find [here](https://huggingface.co/FQiao/GenStereo-sd2.1).
-![](teaser_coco.jpg).
+Our codes are based on [GenWarp](https://github.com/sony/genwarp),  [Moore-AnimateAnyone](https://github.com/MooreThreads/Moore-AnimateAnyone) and other repositories. We thank the authors of relevant repositories and papers.