README.md

---
base_model: Qwen/Qwen2.5-VL-7B-Instruct
library_name: transformers
license: apache-2.0
tags:
- llama-factory
- full
- generated_from_trainer
- spatial-intelligence
- 3d-vision
pipeline_tag: video-text-to-text
model-index:
- name: Proxy3D-8B
  results: []
---

# Proxy3D-8B

[**Proxy3D: Efficient 3D Representations for Vision-Language Models via Semantic Clustering and Alignment**](https://huggingface.co/papers/2605.08064)

Proxy3D-8B is a vision-language model (VLM) specialized in 3D scene understanding and spatial reasoning. It is a fine-tuned version of [Qwen/Qwen2.5-VL-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct) using the **Proxy3D** method, which produces compact yet comprehensive 3D proxy representations for the vision modality to overcome the limitations of standard 2D pipelines.

- **Paper:** [arXiv:2605.08064](https://huggingface.co/papers/2605.08064)
- **Project Page:** [wzzheng.net/Proxy3D](https://wzzheng.net/Proxy3D)
- **GitHub Repository:** [Spacedreamer2384/Proxy3D](https://github.com/Spacedreamer2384/Proxy3D)
- **Dataset:** [SpaceSpan-318K](https://huggingface.co/datasets/Spacewanderer8263/Proxy3D-SpaceSpan-318K)

## Model Description

Spatial intelligence in vision-language models (VLMs) is crucial for reasoning in 3D environments. Proxy3D addresses this by extracting scene features using semantic and geometric encoders from video frames, then performing semantic-aware clustering to obtain a set of proxies in 3D space. 

By utilizing these compact proxy representations, the model achieves state-of-the-art performance in 3D visual question answering (VQA), visual grounding, and general spatial intelligence benchmarks while maintaining high efficiency.

## Training Procedure

The model was trained using a four-stage progressive iterative pipeline to develop spatial reasoning skills, ranging from initial image-text alignment to complex 3D reasoning on the **SpaceSpan** dataset.

### Training Hyperparameters

The following hyperparameters were used during training:
- **Learning rate:** 5e-06
- **Train batch size:** 8
- **Total train batch size:** 128
- **Optimizer:** adamw_torch (betas=(0.9,0.999), epsilon=1e-08)
- **LR scheduler type:** cosine
- **LR scheduler warmup ratio:** 0.1
- **Number of epochs:** 1.0

### Framework Versions

- Transformers 4.55.0
- Pytorch 2.6.0+cu118
- Datasets 3.1.0
- Tokenizers 0.21.1

## Usage

Running this model requires a specific environment setup and custom configuration files to handle the `Qwen2VLBEVForConditionalGeneration` architecture. Please refer to the [Setup section of the GitHub repository](https://github.com/Spacedreamer2384/Proxy3D#%EF%B8%8F-setup) for detailed instructions on how to install and run inference.

## Citation

If you find Proxy3D useful for your research, please cite:

```bibtex
@article{proxy3d2026,
  title={Proxy3D: Efficient 3D Representations for Vision-Language Models via Semantic Clustering and Alignment},
  author={Jiang, Jerry and Sun, Haowen and Gudovskiy, Denis and Nakata, Yohei and Okuno, Tomoyuki and Keutzer, Kurt and Zheng Wenzhao},
  journal={arXiv preprint arXiv:2605.08064},
  year={2026}
}
```

## Acknowledgements

This work builds upon several excellent repositories, including [Qwen2.5-VL](https://github.com/QwenLM/Qwen2.5-VL), [LLaMAFactory](https://github.com/hiyouga/LLaMAFactory), and [GPT4Scene](https://github.com/Qi-Zhangyang/GPT4Scene-and-VLN-R1).