metadata
license: apache-2.0
library_name: transformers
tags:
- vision-encoder
- distillation
- video-language
- siglip2
- dinov3
VisionEncoder Checkpoints
Final model checkpoints from the VisionEncoder research project.
Training code: https://github.com/xiaomoguhz/VisionEncoder
Contents
Each directory corresponds to one training pipeline in the code repo:
| Directory | Training code |
|---|---|
declip_siglip2/spatial_align/ |
declip_siglip2/ — DeCLIP spatial alignment distillation on SigLIP2 using DINOv2 / DINOv3 as teacher |
kd_mllm/s1_kd_pretrain/ |
ms-swift/kd_mllm/ stage-1 pretrain (ms-swift/run_s1.sh) |
kd_mllm/s1_siglip2_qwen3_4b/ |
ms-swift/kd_mllm/ stage-1, SigLIP2 + Qwen3-4B backbone |
kd_mllm/s2_siglip2_qwen3_4b_10pct/ |
ms-swift/kd_mllm/ stage-2 SFT on 10% data (run_s2.sh) |
self_refine/qwen3vl_2b_10pct/ |
ms-swift/self_refine/ — register token injection + auto-calibrated GP threshold loss |
video_mllm_swift/s1_siglip2_qwen3_1.7b/ |
ms-swift/video_mllm/ stage-1 with SigLIP2 encoder |
video_mllm_swift/s1_declip_siglip2_qwen3_1.7b/ |
ms-swift/video_mllm/ stage-1 with DeCLIP-SigLIP2 encoder |
video_mllm_swift/s2_siglip2_qwen3_1.7b_10pct/ |
ms-swift/video_mllm/ stage-2 SFT, SigLIP2 |
video_mllm_swift/s2_declip_siglip2_qwen3_1.7b_10pct/ |
ms-swift/video_mllm/ stage-2 SFT, DeCLIP-SigLIP2 |
video_mllm_swift/s2_image_only_10pct/ |
Ablation: image-only stage-2 training |
ms-swift-data/ |
Not a checkpoint — preprocessed SFT training data (ms-swift/data/) used by the pipelines above |
Related repositories
- Code: https://github.com/xiaomoguhz/VisionEncoder
- Evaluation data (~323 GB tarballs): https://huggingface.co/datasets/xiaomoguhzz/R3-Bench-data