| --- |
| license: apache-2.0 |
| base_model: |
| - Qwen/Qwen3.6-27B |
| --- |
| ## Model details |
|
|
| This model is an INT4 quantized version of the original **Qwen3.6-27B** model. |
|
|
| Quantization was performed with **AutoRound** using a calibration dataset derived from Pile-10k. The calibration set consists of 256 randomly selected samples from diverse categories, with sequence length 2048 and 300 optimization iterations. |
|
|
| The launch parameters below were tested on a headless RTX 3090 using the latest vLLM nightly build. MTP and prefix caching are confirmed to work. |
|
|
| ## vLLM launch example |
|
|
| ```bash |
| vllm serve acyildirimer/Qwen3.6-27B-int4-AutoRound \ |
| --host 0.0.0.0 \ |
| --port 8000 \ |
| --dtype auto \ |
| --kv-cache-dtype auto \ |
| --max-model-len auto \ |
| --reasoning-parser qwen3 \ |
| --enable-auto-tool-choice \ |
| --tool-call-parser qwen3_coder \ |
| --gpu-memory-utilization 0.96 \ |
| --enable-prefix-caching \ |
| --max-num-seqs 1 \ |
| --served-model-name Qwen3.6-27B \ |
| --language-model-only \ |
| --performance-mode interactivity \ |
| --attention-backend auto \ |
| --max-num-batched-tokens 4096 \ |
| --generation-config auto \ |
| --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"repetition_penalty":1.0}' \ |
| --speculative-config '{"method": "mtp", "num_speculative_tokens": 2}' |
| ``` |
|
|
| ## Compatibility |
|
|
| - The launch command was tested with the latest vLLM nightly build. |
| - It should also work with the stable vLLM release, provided that the selected KV cache type is supported. If needed, use a supported `--kv-cache-dtype` value or set it to `auto`. |
| - `turboquant` is not currently working on Ampere GPUs, so it is not enabled in the launch example above. |
|
|
| ## Troubleshooting |
|
|
| - If you encounter out-of-memory errors, reduce `--gpu-memory-utilization`. For example, try `0.94`, `0.92`, or lower depending on your available VRAM and workload. |
| - In some runs, vLLM may incorrectly calculate `Available KV cache memory` during startup. If this happens, terminate the running instance and launch it again with the same parameters. |
|
|
| ## Citation |
|
|
| Please cite both the original Qwen3.6-27B model and the AutoRound quantization method when using this quantized model. |
|
|
| ### Base model |
|
|
| ```bibtex |
| @misc{qwen36-27b, |
| title = {{Qwen3.6-27B}: Flagship-Level Coding in a {27B} Dense Model}, |
| author = {{Qwen Team}}, |
| year = {2026}, |
| month = {April}, |
| url = {https://qwen.ai/blog?id=qwen3.6-27b} |
| } |
| ``` |
|
|
| ### Quantization method |
|
|
| ```bibtex |
| @article{cheng2023optimize, |
| title = {Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs}, |
| author = {Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi}, |
| journal = {arXiv preprint arXiv:2309.05516}, |
| year = {2023} |
| } |
| ``` |
