Update README.md

README.md

@@ -1,3 +1,154 @@
----
-license: mit
----
+---
+license: mit
+---
+<p align="center">
+    <img src="assets/v2sam-logo.png" alt="Image" width="70">
+</p>
+<div align="center">
+<h1 align="center">
+V²-SAM: Marrying SAM2 with Multi-Prompt Experts for Cross-View Object Correspondence
+</h1>
+<h4 align="center"><em>Jiancheng Pan*,     Runze Wang*,     Tianwen Qian,     Mohammad Mahdi,     Xiangyang Xue,</em></h4>
+<h4 align="center"><em>Xiaomeng Huang,     Luc Van Gool,     Danda Pani Paudel,     Yuqian Fu✉ </em></h4>
+<p align="center">
+    <img src="assets/ins.png" alt="Image" width="350">
+</p>
+
+\* *Equal Contribution* &nbsp; &nbsp; Corresponding Author ✉
+</div>
+
+<p align="center">
+    <a href="https://arxiv.org/abs/2511.20886"><img src="https://img.shields.io/badge/Arxiv-2511.20886-b31b1b.svg?logo=arXiv"></a>
+    <a href="https://arxiv.org/abs/2511.20886"><img src="https://img.shields.io/badge/CVPR'26-Paper-blue"></a>
+    <a href="https://huggingface.co/jaychempan/V2-SAM"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Model-HuggingFace-yellow?style=flat&logo=hug"></a>
+    <a href="https://jianchengpan.space/projects/V2-SAM/"><img src="https://img.shields.io/badge/V2--SAM-Project_Page-green"></a>
+    <a href="https://github.com/jaychempan/V2SAM/blob/main/LICENSE"><img src="https://img.shields.io/badge/License-MIT-yellow"></a>
+</p>
+
+<p align="center">
+  <a href="#news">News</a> |
+  <a href="#abstract">Abstract</a> |
+  <a href="#dataset">Dataset</a> |
+  <a href="#model">Model</a> |
+  <a href="#statement">Statement</a>
+</p>
+
+
+## News
+- [2026/2/21] Our V²-SAM is accepted by CVPR 2026. Thanks to all contributors.
+- [2025/11/25] Our paper of "V²-SAM: Marrying SAM2 with Multi-Prompt Experts for Cross-View Object Correspondence" is up on [arXiv](https://arxiv.org/abs/2511.20886).
+
+## Abstract
+Cross-view object correspondence, exemplified by the representative task of ego-exo object correspondence, aims to establish consistent associations of the same object across different viewpoints (e.g., ego-centric and exo-centric). This task poses significant challenges due to drastic viewpoint and appearance variations, making existing segmentation models, such as SAM2, non-trivial to apply directly. To address this, we present V2-SAM, a unified cross-view object correspondence framework that adapts SAM2 from single-view segmentation to cross-view correspondence through two complementary prompt generators. Specifically, the Cross-View Anchor Prompt Generator (V2-Anchor), built upon DINOv3 features, establishes geometry-aware correspondences and, for the first time, unlocks coordinate-based prompting for SAM2 in cross-view scenarios, while the Cross-View Visual Prompt Generator (V2-Visual) enhances appearance-guided cues via a novel visual prompt matcher that aligns ego-exo representations from both feature and structural perspectives. To effectively exploit the strengths of both prompts, we further adopt a multi-expert design and introduce a Post-hoc Cyclic Consistency Selector (PCCS) that adaptively selects the most reliable expert based on cyclic consistency. Extensive experiments validate the effectiveness of V2-SAM, achieving new state-of-the-art performance on Ego-Exo4D (ego-exo object correspondence), DAVIS-2017 (video object tracking), and HANDAL-X (robotic-ready cross-view correspondence).
+
+<p align="center">
+    <img src="assets/v2sam-framework.png" alt="Image">
+</p>
+
+## Dataset
+Our method based on Ego-Exo4D (ego-exo object correspondence), DAVIS-2017 (video object tracking), and HANDAL-X (robotic-ready cross-view correspondence).
+
+We provide the processed versions of these datasets on HuggingFace for easy access:
+
+### 🔹 Ego-Exo4D
+- [Train Split](https://huggingface.co/datasets/jaychempan/Ego-Exo4D-Relation-Train)
+- [Test Split](https://huggingface.co/datasets/jaychempan/Ego-Exo4D-Relation-Test)
+
+### 🔹 DAVIS-2017
+- [Dataset Link](https://huggingface.co/datasets/jaychempan/DAVIS)
+
+### 🔹 HANDAL-X
+- [Dataset Link](https://huggingface.co/datasets/jaychempan/HANDAL)
+
+## Model
+### Environment Setup
+
+```
+conda create -n v2sam python=3.10 -y
+conda activate v2sam
+cd ~/projects/V2-SAM
+export LD_LIBRARY_PATH=/opt/modules/nvidia-cuda-12.1.0/lib64:$LD_LIBRARY_PATH
+export PATH=/opt/modules/nvidia-cuda-12.1.0/bin:$PATH
+# conda install pytorch==2.3.1 torchvision==0.18.1 pytorch-cuda=12.1 cuda -c pytorch  -c "nvidia/label/cuda-12.1.0" -c "nvidia/label/cuda-12.1.1"
+pip install torch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 --index-url https://download.pytorch.org/whl/cu121
+
+# pip install mmcv==2.1.0 -f https://download.openmmlab.com/mmcv/dist/cu121/torch2.3/index.html 
+pip install -U openmim
+mim install mmengine
+mim install "mmcv>=2.1.0"
+pip install -r requirements.txt
+pip install prettytable
+
+# use local mmengine for use the thrid party tools
+cd mmengine
+pip install -e .
+```
+
+### SAM2 and DINOV3 weights
+Choose the base model weights to use.
+```
+huggingface-cli download jaychempan/sam2 --local-dir weights/sam2 --include sam2_hiera_large.pt
+
+huggingface-cli download jaychempan/dinov2 --local-dir weights/dinov2 --include dinov2_vitg14_reg4_pretrain.pth
+
+huggingface-cli download jaychempan/dinov3 --local-dir weights/dinov3 --include dinov3_vitl16_pretrain_lvd1689m-8aa4cbdd.pth
+```
+
+
+### Train
+
+```
+bash tools/dist.sh train projects/v2sam/configs/v2sam.py 4
+```
+if `V²-Visual`, rename the project's dir `projects/v2sam_visual` --> `projects/v2sam`
+
+else `V²-Fusion`, rename the project's dir `projects/v2sam_fusion` --> `projects/v2sam`
+
+> Note: `V²-Anchor` no need to train (use sam2 offical decoder checkpoint)
+
+### Test
+
+```
+bash tools/test.sh test projects/v2sam/configs/v2sam.py 4 /path/to/checkpoint
+
+bash tools/test_all.sh test projects/v2sam/configs/v2sam.py 4 /path/to/checkpoint/dir
+```
+
+## Statement
+### Acknowledgement
+
+This project references and uses the following open source models and datasets.
+
+#### Related Open Source Models
+
+- [Sa2VA](https://arxiv.org/abs/2501.04001)
+- [SAM2](https://arxiv.org/abs/2408.00714)
+- [DINOv2](https://arxiv.org/abs/2304.07193)
+- [DINOv3](https://arxiv.org/abs/2508.10104)
+
+#### Related Open Source Datasets
+
+- [Ego-Exo4D Dataset](https://ego-exo4d-data.org/)
+- [DAVIS-2017 Dataset](https://davischallenge.org/davis2017/code.html)
+- [HANDAL-X Dataset](https://nvlabs.github.io/HANDAL/)
+
+
+### Citation
+
+If you are interested in the following work or want to use our dataset, please cite the following paper.
+
+```bibtex
+@inproceedings{pan2026v,
+  title={V$^{2}$-SAM: Marrying SAM2 with Multi-Prompt Experts for Cross-View Object Correspondence},
+  author={Pan, Jiancheng and Wang, Runze and Qian, Tianwen and Mahdi, Mohammad and Fu, Yanwei and Xue, Xiangyang and Huang, Xiaomeng and Van Gool, Luc and Paudel, Danda Pani and Fu, Yuqian},
+  booktitle={CVPR},
+  year={2026}
+}
+
+@inproceedings{fu2025objectrelator,
+  title={Objectrelator: Enabling cross-view object relation understanding across ego-centric and exo-centric perspectives},
+  author={Fu, Yuqian and Wang, Runze and Ren, Bin and Sun, Guolei and Gong, Biao and Fu, Yanwei and Paudel, Danda Pani and Huang, Xuanjing and Van Gool, Luc},
+  booktitle={ICCV},
+  year={2025}
+}
+```