NOVA pipeline: W4A16 | generic | run=qwen25vl3b_w4a16_generic

README.md

@@ -1,68 +1,67 @@
-        ---
-        base_model: Qwen/Qwen2.5-VL-3B-Instruct
-        tags:
-          - quantized
-          - w4a16
-          - robotics
-          - nova-robot
-        pipeline_tag: image-text-to-text
-        language:
-          - en
-        ---
+---
+base_model: Qwen/Qwen2.5-VL-3B-Instruct
+tags:
+  - quantized
+  - w4a16
+  - robotics
+  - nova-robot
+pipeline_tag: image-text-to-text
+language:
+  - en
+---
 
-        # Qwen2.5-VL-3B-Instruct-W4A16-generic
+# Qwen2.5-VL-3B-Instruct-W4A16-generic
 
-        Quantized with the NOVA quantization pipeline on 2026-04-22.
-        Base model: [Qwen/Qwen2.5-VL-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct)
+Quantized with the NOVA quantization pipeline on 2026-04-22.
+Base model: [Qwen/Qwen2.5-VL-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct)
 
-        ## Quantization details
+## Quantization details
 
-        | Parameter | Value |
-        |---|---|
-        | Method | `W4A16` |
-        | Group size | 128 |
-        | Calibration | `generic` |
-        | Ignored modules | `re:.*lm_head, re:.*visual.*` |
-        | Tool | `llm-compressor >= 0.5.1` |
+| Parameter | Value |
+|---|---|
+| Method | `W4A16` |
+| Group size | 128 |
+| Calibration | `generic` |
+| Ignored modules | `re:.*lm_head, re:.*visual.*` |
+| Tool | `llm-compressor >= 0.4.2` |
 
-        ## Benchmark results
+## Benchmark results
 
-        | Metric | Value |
-        |---|---|
-        | Perplexity (wikitext-2, 20 samples) | 20.643 |
-        | OCR sanity check | ✅ PASS |
-        | Tokens / second | 4.7 |
-        | TTFT (exact, prefill only) | 1.2 ms |
-        | TPOT (exact, per output token) | 215.0 ms |
-        | Inference VRAM | 9.57 GB |
-        | Disk size | 4.03 GB |
+| Metric | Value |
+|---|---|
+| Perplexity (wikitext-2, 20 samples) | 20.864 |
+| OCR sanity check | ✅ PASS |
+| Tokens / second | 1.8 |
+| TTFT (exact, prefill only) | 801.9 ms |
+| TPOT (exact, per output token) | 566.3 ms |
+| Inference VRAM | 16.63 GB |
+| Disk size | 8.21 GB |
 
-        > TTFT and TPOT measured with `BaseStreamer` injection for exact
-        > prefill/decode separation. Values are not conflated throughput averages.
+> TTFT and TPOT measured with `BaseStreamer` injection (prompt-skip corrected).
 
-        ## Registry notes
+## Registry notes
 
-        - Pin transformers==4.49.0 if using AutoAWQ (archived May 2025).
+- Pin transformers==4.49.0 if using AutoAWQ (archived May 2025).
 - Use llm-compressor>=0.5.1 for new quantization runs.
 - Projector (model.visual.merger) kept at FP32 — matched by visual.* regex.
 - OCR and bbox grounding regress 5x faster than MMMU under aggressive quant.
 - Keep merger at FP32, not BF16, for best bbox coordinate precision.
 
-        ## Usage
+## Usage
 
-        ```python
-        from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
-        import torch
+```python
+from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
+import torch
 
-        model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
-            "Mohaaxa/Qwen2.5-VL-3B-Instruct-W4A16-generic",
-            torch_dtype=torch.bfloat16,
-            device_map="auto",
-        )
-        processor = AutoProcessor.from_pretrained("Mohaaxa/Qwen2.5-VL-3B-Instruct-W4A16-generic")
-        ```
+model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
+    "Mohaaxa/Qwen2.5-VL-3B-Instruct-W4A16-generic",
+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+)
+processor = AutoProcessor.from_pretrained("Mohaaxa/Qwen2.5-VL-3B-Instruct-W4A16-generic")
+```
 
-        ## Citation
+## Citation
 
-        If you use this model in research, please cite the NOVA project.
-        Pipeline source: `Mohaaxa/nova-quant-pipeline`
+If you use this model in research, please cite the NOVA project.
+Pipeline source: `Mohaaxa/nova-quant-pipeline`

config.json

@@ -39,7 +39,7 @@
         }
       }
     },
-    "format": "pack-quantized",
+    "format": "dense",
     "global_compression_ratio": null,
     "ignore": [
       "visual.blocks.0.attn.qkv",
@@ -208,7 +208,7 @@
     ],
     "kv_cache_scheme": null,
     "quant_method": "compressed-tensors",
-    "quantization_status": "compressed"
+    "quantization_status": "frozen"
   },
   "rms_norm_eps": 1e-06,
   "rope_scaling": {

model-00001-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bacd32c4c662c186301b4f911d7622cc930f67642bff0503a0eb159fe1b7a1ab
+size 4987765320

model-00002-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7c7670baa573e15dce9ed4c493506f9937a1707d564e5575f5fcfcc571f0b26a
+size 3208992064

quant_meta.json

@@ -8,7 +8,7 @@
   ],
   "calibration": "generic",
   "group_size": 128,
-  "ppl": 20.64317496976273,
+  "ppl": 20.86361947953847,
   "baseline_ppl": 19.096206092078926,
   "sanity_passed": true,
   "sanity_output": "QA_TEST_8472"