README.md

---
license: apache-2.0
library_name: transformers
pipeline_tag: video-text-to-text
tags:
- 4DThinker
- dynamic-spatial-reasoning
- vision-language-model
- latent-reasoning
---

# 4DThinker: Thinking with 4D Imagery for Dynamic Spatial Understanding

[**Paper**](https://huggingface.co/papers/2605.05997) | [**Code**](https://github.com/zhangquanchen/4DThinker)

4DThinker is a framework that enables Vision-Language Models (VLMs) to "think with 4D" through dynamic latent mental imagery—internally simulating how scenes evolve within the continuous hidden space. It addresses dynamic spatial reasoning from monocular video by grounding the model in dynamic visual semantics.

This repository contains the trained model checkpoints from Qwen2.5-VL-3B for **4DThinker**.

## Model Structure

```
model/
├── dift/
│   ├── checkpoints/          # DIFT-stage model weights
│   │   ├── model-00001-of-00002.safetensors
│   │   ├── model-00002-of-00002.safetensors
│   │   ├── config.json
│   │   ├── tokenizer.json
│   │   └── ...
│   └── tensorboard/          # DIFT training logs
└── 4drl/
    ├── model-00001-of-00002.safetensors
    ├── model-00002-of-00002.safetensors
    ├── config.json
    ├── tokenizer.json
    ├── trainer_state.json
    └── ...
```

## Models

| Model | Stage | Base Model | Description |
|-------|-------|------------|-------------|
| `dift/checkpoints/` | DIFT | Qwen2.5-VL-3B-Instruct | Supervised with cosine similarity loss on latent visual tokens |
| `4drl/` | 4DRL (GRPO) | DIFT checkpoint | Reinforced with answer-based rewards |

## Special Tokens

Three special tokens are added to the Qwen2.5-VL vocabulary to support latent imagery:

| Token | Description |
|-------|-------------|
| `<|latent_pad|>` | Padding within latent sequences |
| `<|latent_start|>` | Marks start of latent visual token block |
| `<|latent_end|>` | Marks end of latent visual token block |

## Usage

```python
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor

model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "jankin123/4DThinker-3B",
    subfolder="4drl",
    torch_dtype="auto",
    device_map="auto"
)
processor = AutoProcessor.from_pretrained("jankin123/4DThinker-3B", subfolder="4drl")
```

## Citation

```bibtex
@article{4dthinker,
  title={4DThinker: Thinking with 4D Imagery for Dynamic Spatial Understanding},
  author={Zhang, Quanchen and others},
  journal={arXiv preprint arXiv:2605.05997},
  year={2026}
}
```

## License

Apache License 2.0