Update README.md

README.md

@@ -1,3 +1,191 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+library_name: transformers
+---
+
+# GLUS: Global-Local Reasoning Unified into A Single Large Language Model for Video Segmentation
+<div style="text-align: left;">
+  <p>
+    <a href="https://openreview.net/profile?id=~Lang_Lin3">Lang Lin</a>*, 
+    <a href="https://openreview.net/profile?id=~Xueyang_Yu1">Xueyang Yu</a>*, 
+    <a href="https://ziqipang.github.io/">Ziqi Pang</a>*, 
+    <a href="https://yxw.web.illinois.edu/">Yu-Xiong Wang</a>
+  </p>
+</div>
+
+[[`Project Page`](https://glus-video.github.io/)] [[`arXiv`](https://arxiv.org/abs/2504.07962)] [[`GitHub`](https://github.com/GLUS-video/GLUS)]
+
+
+[![arXiv](https://img.shields.io/badge/arXiv-2504.07962-A42C25?style=flat&logo=arXiv&logoColor=A42C25)](https://arxiv.org/abs/2504.07962)
+[![Project](https://img.shields.io/badge/Project-Page-green?style=flat&logo=Google%20chrome&logoColor=green)](https://glus-video.github.io/) 
+
+
+## Overview
+
+**RefVOS in complex scenarios** places high demands on models' video understanding and fine-grained localization capabilities. Recently, numerous models leveraging **MLLM-based** comprehension and reasoning abilities have been proposed to address this challenge. Our **GLUS** advances further along this methodological path.
+
+🚀 **GLUS is principled.** It utilizes global-local reasoning to combine holistic video understanding with detailed frames understanding, unleashing the potential of fine-grained segmentation in complex scenarios.
+
+✨ **GLUS is powerful.** It unifies the methods of memory bank, object contrastive learning and key frame selection to tackle the problems of mask inconsistency and object obfuscation, achieving state-of-the-art performance in complex-scenario RefVOS tasks.
+
+📌 **GLUS is simple.** It elegantly integrates the approach for complex-scenario RefVOS tasks within a single MLLM framework, eliminating the necessity of utilizing other independent modules.
+
+
+## Installation
+```shell
+git clone git@github.com:GLUS-video/GLUS.git && cd GLUS
+pip install -r requirements.txt
+pip install ./model/segment-anything-2
+pip install flash-attn==2.6.2 --no-build-isolation
+```
+
+## Model Zoo
+
+For more convenient following, we provide the checkpoints of GLUS without object contrastive learning.
+
+| Model                           | Training Datasets          | Methods             | Download |  MeViS J\&F | Ref-Youtube-VOS J\&F |
+|--------------------------------------|---------------------------------|--------------|----------|-----------|-----------------------|
+| **GLUS<sup><i>S</i></sup><sub>partial</sub>** | MeViS, Ref-Youtube-VOS          | GLU + MB |  [HuggingFace](https://huggingface.co/Swindl/GLUS-S-partial/tree/main), [ModelScope](https://www.modelscope.cn/models/LangLin/GLUS-S-partial/files)        | 49.5 | 65.2 |
+| **GLUS<sup><i>S</i></sup>**            | MeViS, Ref-Youtube-VOS          | GLU + MB + OC + KFS |  [HuggingFace](https://huggingface.co/Swindl/GLUS-S/tree/main), [ModelScope](https://www.modelscope.cn/models/LangLin/GLUS-S/files)        | 50.3 | 66.6 |
+| **GLUS<sup><i>A</i></sup>**            | + RefDAVIS17, ReVOS, LVVIS      | GLU + MB |  [HuggingFace](https://huggingface.co/Swindl/GLUS-A/tree/main), [ModelScope](https://www.modelscope.cn/models/LangLin/GLUS-A/files)        | 51.3 | 67.3 |
+
+**Notes**: “GLU”: Global-local unification, “MB”: End-to-end memory bank, “OC”: Object contrastive loss, “KFS”: key frame selection.
+      GLUS<sup><i>S</i></sup> refers to the model trained on a subset of existing RefVOS datasets (Mevis and Ref-Youtube-VOS), while GLUS<sup><i>A</i></sup> denotes the model trained on the full set of available datasets.
+
+We recommend to download and store the pretrained weights at ``GLUS_ROOT/checkpoints``.
+
+## Training and Validation
+
+### 1. Data Preparation
+
+Please follow the below architecture to prepare the datasets. We recommend to set ``DATASET_ROOT``  to ``GLUS_ROOT/data``.
+
+1. RefVOS Datasets: [MeViS](https://github.com/henghuiding/MeViS), [Refer-YouTube-VOS](https://codalab.lisn.upsaclay.fr/competitions/3282#participate-get-data), [Ref-DAVIS17](https://github.com/wjn922/ReferFormer/blob/main/docs/data.md).
+2. Reasoning VOS Datasets: [ReVOS](https://github.com/cilinyan/ReVOS-api), [ReasonVOS](https://github.com/showlab/VideoLISA/blob/main/BENCHMARK.md)
+3. Open-Vocabulary Video Instance Segmentation Dataset: [LV-VIS](https://github.com/haochenheheda/LVVIS/tree/main).
+
+<details open>
+<summary> <strong>Datasets Architecture</strong> </summary>
+
+```
+DATASET_ROOT
+├── mevis
+│   ├── train
+│   │   ├── JPEGImages
+│   │   ├── mask_dict.json
+│   │   └── meta_expressions.json
+│   ├── valid
+│   │   ├── JPEGImages
+│   │   └── meta_expressions.json
+│   └── valid_u
+│       ├── JPEGImages
+│       ├── mask_dict.json
+│       └── meta_expressions.json
+├── Refer-YouTube-VOS
+│   ├── meta_expressions
+│   │   ├─�� train/meta_expressions.json
+│   │   └── valid/meta_expressions.json
+│   ├── train
+│   │   ├── JPEGImages
+│   │   └── Annotations
+│   └── valid
+│       └── JPEGImages
+├── DAVIS17
+│   ├── meta_expressions
+│   │   ├── train/meta_expressions.json
+│   │   └── valid/meta_expressions.json
+│   ├── train
+│   │   ├── JPEGImages
+│   │   └── Annotations
+│   └── valid
+│       ├── JPEGImages
+│       └── Annotations
+├── LVVIS
+│   ├── train
+│   │   └── JPEGImages
+│   ├── mask_dict.json
+│   └── meta_expressions.json
+├── ReVOS
+│   ├── JPEGImages 
+│   ├── mask_dict.json             
+│   ├── mask_dict_foreground.json   
+│   ├── meta_expressions_train_.json 
+│   └── meta_expressions_valid_.json 
+├── ReasonVOS
+│   ├── JPEGImages 
+│   ├── Annotations           
+│   ├── meta_expressions.json 
+
+```
+
+</details>
+
+### 2. Model Weights Preparation
+
+Follow the guidance to prepare for the pretrained weights of LISA and SAM-2 for training GLUS:
+
+1. Download the pretrained weights of LISA from [LISA-7B-v1](https://huggingface.co/xinlai/LISA-7B-v1/tree/main).
+2. Download the pretrained weights of SAM-2 from [sam2_hiera_large](https://dl.fbaipublicfiles.com/segment_anything_2/072824/sam2_hiera_large.pt).
+  
+<details>
+<summary> Then organize them in the following architecture: </summary>
+
+```
+WEIGHTS_ROOT
+├── LISA-7B-v1
+└── sam2_hiera_large.pt
+```
+   
+We recommend to set ``WEIGHTS_ROOT`` to ``GLUS_ROOT/checkpoints``.
+
+</details>
+
+### 3. Training
+
+Set the paths in the scripts and then run ``scripts/train_glus_s.sh`` or ``scripts/train_glus_a.sh``. The scripts will automatically start the training, and transform the saved checkpoint into hugging-face format when the training finished.
+
+#### Key Frame Selection
+For the usage of key frame selection, please refer to the [KFS_README](kfs/README.md).
+
+
+### 4. Evaluation
+
+Set the paths, ``val_set`` and ``set_name`` in ``scripts/inference.sh``, and then run it. It will detect the available GPUs firstly and then individually run parallelizable inference on each gpu.
+
+#### Evaluation with Key Frame Selection
+Set the args ``use_kf`` and ``kf_path`` in ``scripts/inference_kf.sh``, and then run it. We provide our json file on Mevis and Refyoutube-VOS for **GLUS<sup><i>S</i></sup>** on the [google drive](https://drive.google.com/drive/folders/1NcjOguZUmal7Xk7rihyhvs5GRK_RzQSO?usp=sharing).
+
+After the masks are generated completely, run the corresponding evalaution python file in ``utils``. You may need to set the groundtruth mask path, predicted mask path and expressions json file path. Please refer to the eval files to see the help on arguments.
+
+An example:
+
+```
+python utils/eval_mevis.py \\
+  --mevis_exp_path=\'$GLUS_ROOT/data/mevis/valid_u/meta_expressions.json\' \\
+  --mevis_mask_path=\'$GLUS_ROOT/data/mevis/valid_u/mask_dict.json\'
+  --mevis_pred_path=\'$GLUS_ROOT/generated\'
+```
+
+Specially, to evaluate the performance on ``Refer-YouTube-VOS Valid`` or ``MeViS Valid`` benchmarks, you may need to submit the predicted masks results following the guidance at [MeViS-Evaluation-Server](https://codalab.lisn.upsaclay.fr/competitions/15094) or [RefYoutube-Evaluation-Server](https://codalab.lisn.upsaclay.fr/competitions/3282).
+
+## Inference and Demo
+
+Please refer to ``demo.ipynb`` to inference on your own videos and referrings.
+
+For more examples, please refer to our [Project Page](https://glus-video.github.io/).
+## Citation
+If you find this work useful in your research, please consider citing:
+```bibtex
+@inproceedings{lin2025glus,
+  title={GLUS: Global-Local Reasoning Unified into A Single Large Language Model for Video Segmentation},
+  author={Lin, Lang and Yu, Xueyang and Pang, Ziqi and Wang, Yu-Xiong},
+  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
+  year={2025}
+}
+```
+## Acknowledgement
+We thank the contributors to the following open-source projects.  Our project is impossible without the inspirations from these excellent researchers.
+* [LISA](https://github.com/dvlab-research/LISA)
+* [SAM2](https://github.com/facebookresearch/sam2)
+* [Mevis](https://github.com/henghuiding/MeViS)
+* [VISA](https://github.com/cilinyan/VISA)