| --- |
| license: apache-2.0 |
| library_name: transformers |
| pipeline_tag: image-text-to-text |
| tags: |
| - block-diffusion |
| - vision-language-action |
| - autonomous-driving |
| - qwen2.5-vl |
| --- |
| |
| # Fast-dDrive |
|
|
| Fast-dDrive is a block-diffusion Vision-Language-Action (VLA) model for |
| end-to-end autonomous driving, built on Qwen2.5-VL-3B. It pairs section-aware |
| structured-diffusion training (SASD) with scaffold-aware speculative decoding |
| (Scaffold Spec) and an optional shared-prefix multi-trajectory inference |
| scaling scheme, and reaches SOTA accuracy on the Waymo Open Dataset |
| End-to-End Driving (WOD-E2E) benchmark at over 200 tokens / second on a |
| single H100. |
|
|
| ## Quick start |
|
|
| ```python |
| import torch |
| from transformers import AutoModelForCausalLM, AutoProcessor |
| |
| MODEL = "Efficient-Large-Model/Fast_dDrive_3B" # or your local clone |
| |
| processor = AutoProcessor.from_pretrained(MODEL, trust_remote_code=True) |
| model = AutoModelForCausalLM.from_pretrained( |
| MODEL, |
| trust_remote_code=True, |
| dtype=torch.bfloat16, |
| ).cuda().eval() |
| |
| # Scaffold Spec (paper canonical, threshold = 0.0) |
| output_ids = model.scaffold_speculative_sample( |
| input_ids=input_ids, |
| attention_mask=attention_mask, |
| pixel_values=pixel_values, |
| image_grid_thw=image_grid_thw, |
| confidence_threshold=0.0, |
| block_size=32, |
| max_new_tokens=512, |
| ) |
| ``` |
|
|
| ## Inference paths |
|
|
| This release exposes three decoding paths as bound methods on the model: |
|
|
| | Method | Description | Threshold | |
| |---|---|---| |
| | `mdm_sample_deep_scaffold` | **Section Diffusion (SD)** — iterative MDM denoising over a pre-filled JSON scaffold | `0.9` | |
| | `scaffold_speculative_sample` | **Scaffold Spec (SS)** — scaffold-aware self-speculative decoding (MDM draft + AR verify per block). Paper canonical. | `0.0` | |
| | `scaffold_spec_with_ss_multi_traj` | **SS multi-rollout** — shared-prefix N-rollout inference scaling on the trajectory section | `0.0` | |
|
|
| > **Important:** `scaffold_speculative_sample` and its multi-traj variant must |
| > be run with `confidence_threshold=0.0` to reproduce the paper numbers. |
| > Running at `0.9` silently degrades both ADE and throughput. |
| |
| ## Headline results — WOD-E2E test set (single H100) |
| |
| | Mode | RFS ↑ | ADE@3s ↓ | ADE@5s ↓ | TPS ↑ | Tok/Step ↑ | |
| |---|---|---|---|---|---| |
| | Scaffold Spec | 7.823 | 1.254 | 2.907 | 210.4 | 4.90 | |
| | + Inference scaling (N=4) | 7.827 | 1.240 | 2.821 | 114.7 | 2.76 | |
| |
| On the WOD-E2E val set, Scaffold Spec runs at 1919 ms / sample (4.1× over the |
| AR baseline); fused with SGLang the same configuration drops to 665 ms / |
| sample at 608.5 TPS — the 11.8× / 12× speedup over AR cited in the paper. |
| |
| ## Files |
| |
| - `modeling.py` — model definition (`Fast_dDriveForConditionalGeneration`) |
| - `configuration.py` — config classes |
| - `section_utils.py` — scaffold construction + section-aligned block index utilities |
| - `generation_utils.py` — the three inference paths, attached to the model class on import |
| - `config.json`, `generation_config.json`, `preprocessor_config.json`, `chat_template.jinja`, tokenizer files — standard HF artifacts |
| - `model-0000{1..4}-of-00004.safetensors` — model weights (4 shards) |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{zhang2026fastddriveefficientblockdiffusionvlm, |
| title={Fast-dDrive: Efficient Block-Diffusion VLM for Autonomous Driving}, |
| author={Kewei Zhang and Jin Wang and Sensen Gao and Chengyue Wu and Yulong Cao and Songyang Han and Boris Ivanovic and Langechuan Liu and Marco Pavone and Song Han and Daquan Zhou and Enze Xie}, |
| year={2026}, |
| eprint={2605.23163}, |
| archivePrefix={arXiv}, |
| primaryClass={cs.CV}, |
| url={https://arxiv.org/abs/2605.23163}, |
| } |
| ``` |
|
|
