z-lab

/

Qwen3.6-27B-DFlash

like 265

Follow

Z Lab 650

It's still under training so probably not as good as final version.
Also using third-party AWQ-quant might affect accuracy.

Yes, it's still under training. And also this model has some sliding window layers with causal attention, I think vLLM currently doesn't support that. I will talk to vLLM folks to support this new feature soon.

Yes, it's still under training. And also this model has some sliding window layers with causal attention, I think vLLM currently doesn't support that. I will talk to vLLM folks to support this new feature soon.

Thanks for the update, does it mean those "sliding window layers" are currently used full-sequence attention caused imperfect performance? Thanks.

@evilperson068 For current vLLM implementation, yes they are still using full attention :( We will update the SGLang DFlash implementation today or tomorrow to support interleaved SWA layers.

Sadly sglang is not supported on strix halo as far as I know.

And because of other things I moved to Gemma 4 and the performance on strix halo is better on llama then on vllm so I really hope for a dflash Gemma 4 26b a4b dflash to so if it’s working. Had good results on qwen3.6 35b but on vllm. I don’t know if llama supports this draft setup overall.

I'm really looking forward to the complete VLLM update!

@evilperson068 For current vLLM implementation, yes they are still using full attention :( We will update the SGLang DFlash implementation today or tomorrow to support interleaved SWA layers.

Wow that'd be cool to switch to sliding window, could improve the performance (both speed and accuracy) for a lot! Thanks!

Can't wait for the sliding window support in vLLM. That and the updated, better trained model from you guys. Thank you so much.

I used it on Omlx,but it showed thinking as content.

Update: This model has just been updated 9 hours ago, please try the new version people! <3

@fouvy

    --speculative-config '{"method": "dflash", "model": "/path_to/Qwen3.6-27B-DFlash", "num_speculative_tokens": 15}' \

Per-position acceptance rate: 0.685, 0.430, 0.255, 0.107, 0.054, 0.020, 0.013, 0.013, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 10.5%

Your num_speculative_tokens is way too high!

Look at the per-position acceptance, the last 7 positions are 0.000 - you are over-saturating the drafter model. Even those .0.013 are kinda insignificant.

Start with 4 then tune up or down to where you get high Avg. Draft acceptance rate vs accepted throughput that satisfies your needs (actual tok/s speed you desire). It always ends up being a trade-off which greatly depends on the context as well.

vllm: 0.20.0, Qwen3.6-27B-DFlash updated:

Tested cases for 1 request:

"num_speculative_tokens": 2 --> Avg Draft acceptance rate: 71.9% --> tps: 56.89
"num_speculative_tokens": 5 --> Avg Draft acceptance rate: 46.3% --> tps: 62.49
"num_speculative_tokens":10 --> Avg Draft acceptance rate: 20.2% --> tps: 43.82

Those are all slower than mtp: 2 -> tps: 70

@fouvy @mancub Could you try the latest commit in this PR? uv pip install -U --torch-backend=auto "vllm @ git+https://github.com/vllm-project/vllm.git@refs/pull/40898/head"

The vLLM main branch doesn't support the causal SWA layers of this draft model, which will lead to low acceptance rate. This PR fixed this.

@fouvy @mancub Could you try the latest commit in this PR? uv pip install -U --torch-backend=auto "vllm @ git+https://github.com/vllm-project/vllm.git@refs/pull/40898/head"

The vLLM main branch doesn't support the causal SWA layers of this draft model, which will lead to low acceptance rate. This PR fixed this.

Thanks for the update. I just updated the PR, coding performance around 60 TPS maxed out, but reasoning and other text generation speed not accelerated too much. It might need to take more training to surpass native MTP performance.

@fouvy @mancub Could you try the latest commit in this PR? uv pip install -U --torch-backend=auto "vllm @ git+https://github.com/vllm-project/vllm.git@refs/pull/40898/head"

The vLLM main branch doesn't support the causal SWA layers of this draft model, which will lead to low acceptance rate. This PR fixed this.

My launch script on 2x3090. After patched your new pr, performance almost the same.

export CONTEXT_LENGTH=166144

export CUDA_DEVICE_WAIT_POLICY=YIELD
export VLLM_SLEEP_WHEN_IDLE=1

CUDA_DEVICE_ORDER=PCI_BUS_ID \
CUDA_VISIBLE_DEVICES=0,1 \
VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 \
VLLM_USE_MODELSCOPE=true vllm serve \
        /path_to/Qwen3.6-27B-AWQ \
        --port 8000 \
        --tensor-parallel-size 2 \
        --max-model-len $CONTEXT_LENGTH \
        --served-model-name qwen-medium \
        --enable-auto-tool-choice \
        --max-num-seqs 10 \
        --gpu-memory-utilization 0.98 \
        --reasoning-parser qwen3 \
        --enable-prefix-caching \
        --kv-cache-dtype auto \
        --speculative-config '{"method": "dflash", "model": "/path_to/Qwen3.6-27B-DFlash", "num_speculative_tokens": 5}' \
        --tool-call-parser qwen3_coder

Just tested vllm nightly + the SWA PR, here are my results on 2x RTX PRO 6000 running on Qwen3.6-27B BF16. Performance degrades at num_speculative_tokens 4 and 16.
It seems that at higher concurrency (C=4) MTP outperforms DFlash.

DFlash = 8

Prefill tok/s                         Aggregate decode tok/s
╭──────┬────────┬────────┬───────┬───╮╭────────────┬───────┬───────┬───────╮
│ ctx  │ tokens │ TTFT s │ tok/s │ N ││ ctx \ conc │     1 │     2 │     4 │
├──────┼────────┼────────┼───────┼───┤├────────────┼───────┼───────┼───────┤
│ 8k   │  5,333 │   0.76 │ 7,059 │ 1 ││ 0          │ 103.2 │ 209.4 │ 377.3 │
│ 16k  │ 10,517 │   1.52 │ 6,941 │ 1 ││ 16k        │ 108.8 │ 211.3 │ 372.0 │
│ 32k  │ 20,887 │   3.05 │ 6,840 │ 1 ││ 32k        │ 105.2 │ 193.7 │ 392.9 │
│ 64k  │ 41,626 │   6.41 │ 6,491 │ 1 ││ 64k        │ 115.0 │ 193.7 │ 360.2 │
│ 128k │ 83,076 │  13.97 │ 5,946 │ 1 ││ 128k       │ 101.9 │ 180.3 │ 318.7 │
╰──────┴────────┴────────┴───────┴───╯╰────────────┴───────┴───────┴───────╯

MTP = 3

Prefill tok/s                         Aggregate decode tok/s
╭──────┬────────┬────────┬───────┬───╮╭────────────┬───────┬───────┬───────╮
│ ctx  │ tokens │ TTFT s │ tok/s │ N ││ ctx \ conc │     1 │     2 │     4 │
├──────┼────────┼────────┼───────┼───┤├────────────┼───────┼───────┼───────┤
│ 8k   │  5,334 │   0.78 │ 6,876 │ 1 ││ 0          │  96.0 │ 189.1 │ 391.5 │
│ 16k  │ 10,518 │   1.54 │ 6,816 │ 1 ││ 16k        │ 100.6 │ 194.5 │ 388.6 │
│ 32k  │ 20,888 │   3.12 │ 6,697 │ 1 ││ 32k        │  97.2 │ 192.4 │ 384.0 │
│ 64k  │ 41,627 │   6.57 │ 6,340 │ 1 ││ 64k        │  98.6 │ 190.3 │ 370.4 │
│ 128k │ 83,077 │  14.34 │ 5,792 │ 1 ││ 128k       │  92.3 │ 178.0 │ 345.0 │
╰──────┴────────┴────────┴───────┴───╯╰────────────┴───────┴───────┴───────╯

I tried on DGX spark and I get virtually 0% acceptance for 4th and subsequent token with FP8 version of Qwen3.6-27B and significantly better results (positive acceptance for up to 8-9 tokens ahead) with BF16 version of the model so base quantization seems to matter a lot. The slowdown from going to BF16 seems to not justify using DFlash currently though.