Create README.md

README.md

@@ -0,0 +1,224 @@
+---
+license: mit
+library_name: transformers
+pipeline_tag: image-text-to-text
+language:
+- en
+- zh
+tags:
+- Bard-VL
+- MLLM
+- vision-language
+- multimodal
+- discrete-diffusion
+- masked-decoding
+- custom_code
+metrics:
+- accuracy
+---
+
+<h1 align="center">BARD: Bridging AutoRegressive and Diffusion Vision-Language Models Via Highly Efficient Progressive Block Merging and Stage-Wise Distillation</h1>
+
+<p align="center">
+  <a href="https://github.com/cbyzju">Baoyou Chen</a><sup>1,3</sup> ·
+  <a href="https://github.com/1ring2rta">Hanchen Xia</a><sup>1</sup> ·
+  <a href="https://github.com/yhpengtu-rgb">Peng Tu</a><sup>1</sup> ·
+  <a href="https://github.com/Theseus-427">Haojun Shi</a><sup>1</sup> ·
+  <a href="https://github.com/AricGamma">Liwei Zhang</a><sup>1</sup> ·
+  <a href="https://github.com/weihaosky">Weihao Yuan</a><sup>4</sup> ·
+  <a href="https://sites.google.com/site/zhusiyucs/home">Siyu Zhu</a><sup>1,2,3,†</sup>
+</p>
+
+<p align="center">
+  <sup>1</sup>Shanghai Academy of AI for Science
+  &nbsp;&nbsp;·&nbsp;&nbsp;
+  <sup>2</sup>Shanghai Innovation Institute
+  &nbsp;&nbsp;·&nbsp;&nbsp;
+  <sup>3</sup>Fudan University
+  &nbsp;&nbsp;·&nbsp;&nbsp;
+  <sup>4</sup>Nanjing University
+</p>
+
+<p align="center">
+  🤗 <a href="https://huggingface.co/fudan-generative-ai/Bard-VL-B16-Mask-4B-Distil-Instruct">Model</a>
+  &nbsp;&nbsp;|&nbsp;&nbsp;
+  🏠 <a href="https://fudan-generative-vision.github.io/Bard-VL">Project Page</a>
+  &nbsp;&nbsp;|&nbsp;&nbsp;
+  📑 <a href="https://huggingface.co/papers/2604.16514">Paper</a>
+  &nbsp;&nbsp;|&nbsp;&nbsp;
+  ✨ <a href="https://github.com/fudan-generative-vision/Bard-VL">Code</a>
+</p>
+
+<p align="center">
+  <img src="https://img.shields.io/badge/Params-4B-0ea5e9" alt="Params">
+  <img src="https://img.shields.io/badge/Input-Multimodal-10b981" alt="Input">
+  <img src="https://img.shields.io/badge/Decoding-Masked%20Diffusion-f59e0b" alt="Decoding">
+  <img src="https://img.shields.io/badge/Language-EN%20%7C%20ZH-8b5cf6" alt="Language">
+  <img src="https://img.shields.io/badge/License-MIT-64748b" alt="License">
+</p>
+
+<p align="center">
+  🧠 Multimodal Understanding &nbsp;&nbsp;|&nbsp;&nbsp; 🧩 Blockwise Denoising &nbsp;&nbsp;|&nbsp;&nbsp; ⚡ Diffusion-Style Decoding
+</p>
+
+# Bard-VL-B16-Mask-4B-Distil-Instruct
+
+**Bard-VL-B16-Mask-4B-Distil-Instruct** is a 4B-class distilled multimodal instruction model with **masked discrete-diffusion-style decoding**.
+
+It is part of the **BARD-VL** family and is designed to bridge autoregressive and diffusion-style multimodal large language models through **Progressive Block Merging (PBM)** and **Stage-Wise Distillation (SWD)**.
+
+Compared with a standard autoregressive VLM release style, Bard-VL emphasizes:
+
+- **a distilled design for stronger efficiency-quality tradeoffs under larger block sizes**
+- **masked / diffusion-style decoding**
+- **controllable response generation through blockwise denoising**
+- **multimodal understanding under a unified visual-language interface**
+
+---
+
+## ✨ Highlights
+
+- **Discrete Diffusion MLLM**: Bard-VL replaces standard left-to-right decoding with masked blockwise denoising.
+- **PBM + SWD**: the BARD family combines progressive block merging and stage-wise distillation to bridge AR and diffusion VLMs.
+- **Distilled Variant**: this checkpoint is tuned for a stronger efficiency-quality tradeoff when running with larger block sizes.
+- **Multimodal Understanding**: Bard-VL is built as a general visual-language model rather than a modality-specific demo.
+- **Controllable Decoding**: generation quality and speed can be controlled through `block_size`, `denoising_steps`, and remasking strategy.
+
+---
+
+## 📊 Evaluation Results
+
+### AutoRegressive Vision-Language Models
+
+| Model | Parameters | MMMU<sub>val</sub> | MMMU-Pro<sub>standard</sub> | MME<sub>sum</sub> | RealWorldQA | MMStar | AI2D | ChartQA |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|
+| Qwen3-VL | 4B | 47.9 | 35.0 | 2297 | 70.5 | 56.9 | 81.0 | 80.9 |
+| Qwen3-VL | 8B | 53.0 | 36.0 | 2379 | 69.5 | 59.9 | 83.5 | 84.0 |
+| InternVL3.5 | 4B | 57.4 | 38.2 | 2236 | 66.7 | 65.6 | 80.6 | 86.2 |
+| InternVL3.5 | 8B | 57.2 | 41.0 | 2359 | 63.1 | 66.3 | 82.1 | 87.0 |
+
+### Diffusion Vision-Language Models
+
+| Model | Parameters | MMMU<sub>val</sub> | MMMU-Pro<sub>standard</sub> | MME<sub>sum</sub> | RealWorldQA | MMStar | AI2D | ChartQA |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|
+| LLaDA-V | 8B | 48.8 | 35.4 | 1998 | 63.4 | 60.4 | 77.8 | 78.2 |
+| Dream-VL | 7B | 51.6 | 25.0 | 2179 | 67.7 | 59.9 | 80.4 | 86.2 |
+| LaviDa | 8B | 44.2 | 28.6 | 1711 | 40.3 | 47.0 | 70.1 | 64.6 |
+| SDAR-VL | 8B | 44.0 | 28.2 | 2142 | 66.1 | 53.3 | 79.6 | 82.4 |
+| MMaDA | 8B | 30.2 | 21.5 | 1287 | 28.2 | 25.7 | 54.9 | 43.2 |
+| Dimple-VL | 7B | 46.4 | 24.1 | 1924 | 51.9 | 47.7 | 74.2 | 58.4 |
+
+### BARD-VL Converted from Qwen3-VL
+
+| Model | Parameters | MMMU<sub>val</sub> | MMMU-Pro<sub>standard</sub> | MME<sub>sum</sub> | RealWorldQA | MMStar | AI2D | ChartQA |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|
+| BARD-VL (*B* = 32) | 2B | 42.0 | 27.9 | 2045 | 64.6 | 53.1 | 72.6 | 76.8 |
+| BARD-VL (*B* = 32) | 4B | 53.0 | 34.2 | 2305 | 71.9 | 63.6 | 82.8 | 80.2 |
+| BARD-VL (*B* = 32) | 8B | 54.6 | 37.6 | 2393 | 70.7 | 65.0 | 83.2 | 84.6 |
+
+---
+
+## 🛠️ Environment
+
+Make sure your environment includes versions close to the following:
+
+```bash
+transformers>=4.46
+torch>=2.5
+accelerate>=1.6
+python>=3.10
+```
+
+Recommended runtime settings in the local repository:
+
+```bash
+dtype = bfloat16
+attn_implementation = sdpa
+block_size = 16
+denoising_steps = 16
+```
+
+---
+
+## 🚀 Inference Example
+
+```python
+import torch
+from transformers import AutoProcessor, AutoModel
+from PIL import Image
+
+model_name = "fudan-generative-ai/Bard-VL-B16-Mask-4B-Distil-Instruct"
+
+processor = AutoProcessor.from_pretrained(
+    model_name,
+    trust_remote_code=True,
+)
+
+model = AutoModel.from_pretrained(
+    model_name,
+    torch_dtype=torch.bfloat16,
+    trust_remote_code=True,
+)
+
+messages = [
+    {
+        "role": "system",
+        "content": "You are a helpful assistant.",
+    },
+    {
+        "role": "user",
+        "content": [
+            {"type": "image", "image": "assets/puzzle.jpg"},
+            {"type": "text", "text": "Please describe this image and explain the key visual evidence."},
+        ],
+    },
+]
+
+text = processor.apply_chat_template(
+    messages,
+    tokenize=False,
+    add_generation_prompt=True,
+)
+
+inputs = processor(
+    text=[text],
+    images=[Image.open("assets/puzzle.jpg").convert("RGB")],
+    return_tensors="pt",
+)
+
+input_ids = inputs.pop("input_ids")
+
+outputs = model.generate(
+    input_ids=input_ids,
+    max_new_tokens=512,
+    block_size=16,
+    denoising_steps=16,
+    temperature=0.0,
+    top_k=0,
+    top_p=1.0,
+    remasking_strategy="low_confidence_dynamic",
+    confidence_threshold=0.5,
+    **inputs,
+)
+
+generation = processor.tokenizer.decode(
+    outputs[0][len(input_ids[0]):].tolist(),
+    skip_special_tokens=True,
+)
+
+print(generation)
+```
+
+
+---
+
+## 📚 Citation
+
+```bibtex
+@article{chen2026bard,
+  title={BARD: Bridging AutoRegressive and Diffusion Vision-Language Models Via Highly Efficient Progressive Block Merging and Stage-Wise Distillation},
+  author={Baoyou Chen and Hanchen Xia and Peng Tu and Haojun Shi and Liwei Zhang and Weihao Yuan and Siyu Zhu},
+  journal={arXiv preprint arXiv:2604.16514},
+  year={2026}
+}
+```