Add MIGRATION_GUIDE.md

MIGRATION_GUIDE.md

@@ -0,0 +1,292 @@
+# Migration Guide: StreamingVLM → Qwen3-VL on ROCm
+
+## Overview of Changes
+
+This document details every change made to port StreamingVLM from its original
+implementation (Qwen2.5-VL-7B + flash-attn + CUDA) to (Qwen3-VL-4B + SDPA + ROCm).
+
+---
+
+## 1. Dependency Changes
+
+### Original (`infer_requirements.txt`)
+```
+transformers==4.52.4
+flash_attn==2.8.0.post2
+torch==2.7.1
+liger_kernel==0.6.1
+qwen-vl-utils==0.0.11
+```
+
+### Updated (`requirements.txt`)
+```
+transformers>=4.57.0  (install from source: pip install git+https://github.com/huggingface/transformers)
+torch>=2.4.0          (ROCm or CUDA build)
+qwen-vl-utils>=0.0.14
+# NO flash-attn required!
+```
+
+**Why**: Qwen3-VL requires transformers 4.57.0+ (unreleased, must install from source).
+Flash-attn is completely eliminated — replaced with `torch.nn.functional.scaled_dot_product_attention`.
+
+---
+
+## 2. Model Class Changes
+
+| Original | Updated |
+|----------|---------|
+| `Qwen2_5_VLForConditionalGeneration` | `Qwen3VLForConditionalGeneration` |
+| `from transformers.models.qwen2_5_vl.modeling_qwen2_5_vl import ...` | No private imports needed |
+| `attn_implementation="flash_attention_2"` | `attn_implementation="sdpa"` |
+
+### Loading Code
+
+```python
+# BEFORE (original)
+from transformers import Qwen2_5_VLForConditionalGeneration
+model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
+    model_path, torch_dtype="auto", device_map="cuda",
+    attn_implementation="flash_attention_2"
+)
+
+# AFTER (this port)  
+from transformers import Qwen3VLForConditionalGeneration
+model = Qwen3VLForConditionalGeneration.from_pretrained(
+    "Qwen/Qwen3-VL-4B-Instruct",
+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+    attn_implementation="sdpa"  # Works on ROCm + CUDA
+)
+```
+
+---
+
+## 3. Attention Mechanism Replacement
+
+### Language Model Attention
+
+**Original**: Used `_flash_attention_forward()` (private transformers API, CUDA-only)
+```python
+from transformers.models.qwen2_5_vl.modeling_qwen2_5_vl import _flash_attention_forward
+attn_output = _flash_attention_forward(query, key, value, attention_mask, ...)
+```
+
+**Updated**: Uses `F.scaled_dot_product_attention()` (PyTorch native, ROCm+CUDA)
+```python
+import torch.nn.functional as F
+attn_output = F.scaled_dot_product_attention(
+    query_states, key_states, value_states,
+    attn_mask=attention_mask,
+    dropout_p=0.0,
+    is_causal=(attention_mask is None and q_len > 1),
+)
+```
+
+### Vision Encoder Attention
+
+**Original**: Used `flash_attn_varlen_func` for packed variable-length sequences
+```python
+from flash_attn import flash_attn_varlen_func
+attn_output = flash_attn_varlen_func(q, k, v, cu_seqlens, cu_seqlens, max_seqlen, max_seqlen)
+```
+
+**Updated**: Chunked SDPA — splits at sequence boundaries, applies SDPA per chunk
+```python
+def _sdpa_varlen(query, key, value, cu_seqlens, max_seqlen):
+    outputs = []
+    for i in range(cu_seqlens.shape[0] - 1):
+        start, end = cu_seqlens[i].item(), cu_seqlens[i+1].item()
+        q_i = query[start:end].unsqueeze(0).transpose(1, 2)
+        k_i = key[start:end].unsqueeze(0).transpose(1, 2)
+        v_i = value[start:end].unsqueeze(0).transpose(1, 2)
+        out = F.scaled_dot_product_attention(q_i, k_i, v_i, is_causal=False)
+        outputs.append(out.transpose(1, 2).squeeze(0))
+    return torch.cat(outputs, dim=0)
+```
+
+---
+
+## 4. Architecture Differences (Qwen2.5-VL → Qwen3-VL)
+
+| Feature | Qwen2.5-VL-7B | Qwen3-VL-4B |
+|---------|---------------|--------------|
+| ViT patch size | 14×14 | **16×16** |
+| ViT depth | 32 layers | **24 layers** |
+| ViT hidden | 1280 | **1024** |
+| LM hidden | 3584 | **2560** |
+| LM layers | 28 | **36** |
+| KV heads | 4 | **8** |
+| Max context | 128K | **256K** |
+| RoPE theta | 1M | **5M** |
+| MRoPE | standard | **interleaved** |
+| QK-Norm | ❌ | **✅** |
+| DeepStack | ❌ | **✅ (layers 5,11,17)** |
+| ViT window attn | Yes | **No (full attn)** |
+
+### Impact on StreamingVLM:
+- **QK-Norm**: Added `self.q_norm(query)` and `self.k_norm(key)` before RoPE in attention
+- **DeepStack**: Vision encoder extracts features from intermediate layers for richer representation
+- **Interleaved MRoPE**: `mrope_section=[24,20,20]` (temporal/height/width) with interleaved application
+- **No ViT window attention**: Simpler vision encoder — no `fullatt_block_indexes` logic needed
+
+---
+
+## 5. Monkey-Patch Target Changes
+
+### Original (10 patches on Qwen2.5-VL):
+```python
+model.generate = streaming_generate
+model.prepare_inputs_for_generation = ...
+model._sample = ...
+model.forward = qwen2_5_vl_forward
+model.model.forward = model_forward
+model.model.language_model.forward = streaming_language_model_forward  
+model.model.language_model._update_causal_mask = ...
+for layer in model.model.language_model.layers:  # "language_model" submodule
+    layer.forward = streaming_text_decoder_layer_forward
+    layer.self_attn.forward = streaming_text_flash_attn_forward
+model.model.visual.forward = streaming_visual_encoder_forward
+```
+
+### Updated (simplified patches on Qwen3-VL):
+```python
+model.forward = streaming_qwen3_vl_forward
+model.model.forward = streaming_model_forward
+for layer in model.model.layers:  # directly on model.model (no "language_model"!)
+    layer.forward = streaming_text_decoder_layer_forward
+    layer.self_attn.forward = streaming_text_sdpa_forward
+model.model.visual.forward = streaming_visual_encoder_forward
+for blk in model.model.visual.blocks:
+    blk.forward = streaming_visual_block_forward
+    blk.attn.forward = streaming_visual_attention_forward
+model.model.get_rope_index = get_rope_index_streaming
+```
+
+**Key structural difference**: In Qwen2.5-VL, text layers are at `model.model.language_model.layers`.
+In Qwen3-VL, they are at `model.model.layers` (no intermediate `language_model` wrapper).
+
+---
+
+## 6. KV Cache API Changes
+
+### transformers 4.50-4.52 (original):
+```python
+cache.key_cache[layer_idx]   # List[Tensor]
+cache.value_cache[layer_idx]
+```
+
+### transformers 5.8.0+ (updated):
+```python
+cache.layers[layer_idx].keys   # DynamicLayer objects
+cache.layers[layer_idx].values
+```
+
+Our `StreamingCache` handles both via `_get_layer_keys()`/`_set_layer_keys()` compat methods.
+
+---
+
+## 7. Token IDs (Verified for Qwen3-VL)
+
+| Token | ID | Same as 2.5? |
+|-------|-----|--------------|
+| `<\|im_start\|>` | 151644 | ✅ |
+| `<\|im_end\|>` | 151645 | ✅ |
+| `<\|vision_start\|>` | 151652 | ✅ |
+| `<\|vision_end\|>` | 151653 | ✅ |
+| `<\|image_pad\|>` | 151655 | ✅ |
+| `<\|video_pad\|>` | 151656 | ✅ |
+
+Token IDs are identical — no changes needed for token-level logic.
+
+---
+
+## 8. ROCm-Specific Configuration
+
+### Performance tiers (recommended):
+
+| Priority | Implementation | Install | Performance | Platform |
+|----------|---------------|---------|-------------|----------|
+| 1 | `flash_attention_2` + CK backend | Build flash-attn from source | Best | ROCm ≥5.7 |
+| 2 | `sdpa` (default) | None (PyTorch built-in) | Good | Any |
+| 3 | `eager` | None | Slowest | Any |
+
+### For ROCm flash-attn (optional, for maximum throughput):
+```bash
+git clone https://github.com/Dao-AILab/flash-attention.git
+cd flash-attention
+pip install .
+# Then use: attn_implementation="flash_attention_2"
+```
+
+### Environment variables:
+```bash
+export PYTORCH_ROCM_ARCH="gfx942"    # MI300X
+# export PYTORCH_ROCM_ARCH="gfx90a"  # MI250X
+# export PYTORCH_ROCM_ARCH="gfx1100" # RX 7900 XTX
+
+# Flash-attn backend selection:
+export FLASH_ATTENTION_TRITON_AMD_ENABLE="FALSE"  # CK (default, recommended)
+# export FLASH_ATTENTION_TRITON_AMD_ENABLE="TRUE" # Triton (alternative)
+```
+
+---
+
+## 9. Eliminated Private API Dependencies
+
+The original code imported many private symbols from transformers internals.
+These are **all eliminated** in this port:
+
+```python
+# REMOVED — These broke between transformers versions:
+from transformers.models.qwen2_5_vl.modeling_qwen2_5_vl import (
+    _flash_attention_forward,      # Private, CUDA-only
+    flash_attn_varlen_func,        # Private re-export of flash_attn
+    rotate_half,                   # Reimplemented locally
+    repeat_kv,                     # Reimplemented locally
+    apply_rotary_pos_emb_flashatt, # Not needed with SDPA
+    StaticCache,                   # Using DynamicCache directly
+    SlidingWindowCache,            # Not used
+    AttentionMaskConverter,        # Not used
+    make_flex_block_causal_mask,   # Not used
+    BlockMask,                     # Not used
+)
+
+# KEPT — Only stable public APIs:
+from transformers import (
+    Qwen3VLForConditionalGeneration,  # Public model class
+    AutoProcessor,                     # Public processor
+    DynamicCache,                      # Public cache class
+)
+from transformers.modeling_outputs import (
+    BaseModelOutputWithPast,           # Public output type
+    CausalLMOutputWithPast,            # Public output type
+)
+```
+
+---
+
+## 10. Running the Updated Code
+
+### Inference (streaming commentary):
+```bash
+python -m streaming_vlm.inference.inference \
+    --model_path Qwen/Qwen3-VL-4B-Instruct \
+    --video_path match.mp4 \
+    --output_path commentary.vtt \
+    --fps 2 \
+    --attn_implementation sdpa
+```
+
+### Training (Stage 1):
+```bash
+# Download data
+# huggingface-cli download mit-han-lab/Inf-Stream-Train --local-dir ./data
+
+bash scripts/sft_stage_1.sh
+```
+
+### Tests:
+```bash
+python test_imports.py
+# Expected: 6/6 tests pass with no CUDA/flash-attn dependency
+```