| --- |
| license: cc-by-nc-4.0 |
| task_categories: |
| - image-segmentation |
| tags: |
| - glass-surface-detection |
| - rgb-d |
| - scene-understanding |
| - pytorch |
| pretty_name: RGBD-GSD-Net (RGB-D Glass Surface Detection Network) |
| --- |
| |
| # RGBD-GSD-Net — RGB-D Glass Surface Detection Network |
|
|
| Pre-trained weights for the model introduced in: |
|
|
| > **Leveraging RGB-D Data with Cross-Modal Context Mining for Glass Surface Detection** |
| > Jiaying Lin\*, Yuen-Hei Yeung\*, Shuquan Ye, Rynson W. H. Lau |
| > AAAI 2025 |
| > [arXiv](https://arxiv.org/abs/2206.11250) · [Project Page](https://jiaying.link/aaai2025-rgbdglass/) · [Dataset (RGBD-GSD)](https://huggingface.co/datasets/garrying/RGBD-GSD) |
|
|
| ## Model Summary |
|
|
| RGBD-GSD-Net detects glass surfaces by jointly processing RGB images and depth maps. It introduces two novel modules: |
|
|
| - **Cross-Modal Context Mining (CCM)**: adaptively learns individual and mutual context features from RGB and depth information. |
| - **Depth-Missing Aware Attention (DAA)**: explicitly exploits spatial locations where depth is missing (a strong indicator of glass surfaces) to guide detection. |
|
|
| The backbone is a ResNeXt encoder shared across both modalities. |
|
|
| | File | Description | |
| |------|-------------| |
| | `best.pth` | Best checkpoint (204 MB), saved as `{'model': state_dict, ...}` | |
| | `results/our_best_results.zip` | Model predictions on the RGBD-GSD test set | |
|
|
| ## Loading the Weights |
|
|
| ```python |
| import torch |
| from networks.your_network import RGBDGlassNet # from the code release |
| |
| model = RGBDGlassNet() |
| checkpoint = torch.load("best.pth", map_location="cpu") |
| model.load_state_dict(checkpoint["model"]) |
| model.eval() |
| ``` |
|
|
| Download the checkpoint: |
| ```bash |
| huggingface-cli download garrying/RGBD-GSD-Net best.pth --local-dir ./weights |
| ``` |
|
|
| ## Training Dataset |
|
|
| This model was trained and evaluated on **RGBD-GSD**, the first large-scale RGB-D glass surface detection dataset: |
| - 3,009 RGB-D images with binary glass surface masks and depth maps |
| - Available at [garrying/RGBD-GSD](https://huggingface.co/datasets/garrying/RGBD-GSD) |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{aaai2025_rgbdglass, |
| author = {Lin, Jiaying and Yeung, Yuen-Hei and Ye, Shuquan and Lau, Rynson W.H.}, |
| title = {Leveraging RGB-D Data with Cross-Modal Context Mining for Glass Surface Detection}, |
| journal = {AAAI}, |
| year = {2025}, |
| } |
| ``` |
|
|
| ## License |
|
|
| Non-commercial use only — [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/). |
|
|
