Hi, I wonder if that's because the rest of the model is in Q8. Usually you're memory bandwidth bound though so I wouldn't expect it to be that drastically slower.

These have been updated with the fused up+gate tensors and maybe your backend doesn't have optimizations for that yet?

Regarding the imatrix, I use @bartowski 's calibration_datav5 which is a general purpose dataset.

It may certainly be because of the 8bpw, so basically with your method used, you allow the user higher precision (nearly FP) with cheaper hardware.
Would be great to find a way to increase the speed (I will see if I can do anything about it). This is the right direction, you nailed it.
My backend is updated daily, so that´s not the issue. Care to share how you load it (--flags and cmd prompt).
If possible to share also your method scripts would be awesome ,a huge contribution to me and to the community, (The method already shared is a bit unclear).

Thanks in advance.

Hi, great job btw with the weights but it is still 3-4 times slower than the average quantization, we trading speed for accuracy.
Which dataset did you use for the Imatrix file, (is it a coding one or general knowledge)?

Care to elaborate:
A)
What kind of rig you have?
B)
Which (Qwen3.5-397B-A17B) version you're talking here, being 3-4 times slower and vs what.

Care to elaborate:
A)
What kind of rig you have?
B)
Which (Qwen3.5-397B-A17B) version you're talking here, being 3-4 times slower and vs what.

Sure, independently of what kind of hardware used, the difference of inference speed should remain constant relatively to the hardware, using same models, same loading parameters and same backend.
I am using HugstonOne Enterprise Edition 1.0.9 with the last updated qwen3.5-397B-q4_k_m (4 days ago from AeSsedai) which runs around 1tps, compared to the other q4_k_m myself quantized or available in huggingface which run to 3-5tps. Unsloth run faster but in my test I got mostly lower accuracy, which is good for general purposes but useless to us because we using it for very high precision tasks. That´s why AesSedai offer a good solution keeping compression and conserving accuracy.
What´s left obviously is to understand better the method so to possibly optimize the inference speed. For that hopefully AeSsedai hopefully will be willing to share his method here.
Edit: if you have any suggestions or see that I am missing something you are welcome to reply :)

@Trilogix1 I've shared my method previously, it basically looks like what I described here: https://huggingface.co/ubergarm/Qwen3.5-122B-A10B-GGUF/discussions/2#699e4ad09b7edc25c095a595

If you look at the table on the model card for this model, I have a "mixture" column which is in Default Type / FFN Up / FFN Gate / FFN Down notational format. So you can see the Q4_K_M uses Q8_0 as the default type (so that's shared expert, attention, etc.) and I use Q4_K for both the FFN Up and FFN Gate and I use Q5_K for the FFN Down tensors. I also use F32 for most of the SSM tensors because 1) they're tiny and 2) they seem very prone to quantization error based on some feedback I've received.

TG speed should mostly be dominated by memory bandwidth, not compute usually, so I'm a bit confused as to why my quants are slower to run. The overall size is comparable. I need to download eg unsloth's Q4_K_M and do a tensor-quantization comparison and see if I can narrow down the culprit at some point.

@AesSedai > download eg unsloth's Q4_K_M and do a tensor-quantization comparison and see if I can narrow down the culprit at some point.
It´s definitely worth checking, I see it as a game changer with great benefits.

@Trilogix1 I did look into it a bit last night and it turns out their Q4_K_M is nearly identical to mine with three differences:

1. I use the fused Up + Gate and they use the normal split ffn_up_exps and ffn_gate_exps
2. For the output tensor I use Q8_0 and they use Q6_K
3. For the ssm_alpha and ssm_beta tensors I use F32 and they use Q8_0.

I wasn't aware before looking into this that they've adopted @ddh0 's and my MoE-quantization schema. I thought their non-UD quants were the regular llama.cpp variety but they've changed it up quietly.

The overall size change due to those three tensors (output, ssm_alpha, and ssm_beta) is less than 1.5GiB so it's not a size difference. It might be due to the fused Up + Gate. What backend are you running this on? Because I know that CUDA has some supported ops that were added for that fused tensor and I'm wondering if another backend is falling back to some default op instead.

@Trilogix1

still 3-4 times slower than the average quantization

Are you talking about prompt processing or token generation speed here?

In general PP is compute bottlenecked and TG is memory bandwidth bottlenecked.

I did a quick 3-way comparison with some of my test quants that are identical except for the ssm_(alpha|beta) tensors are either:

• q8_0
• bf16 (the native original tensor type)
• f32 (upcast just for speed purposes)

I did not test f16 as that is a downcast and could lead to potential clipping and issues and should be avoided in almost every case.

tl;dr;

They are all very similar in speed and quality as measured by PPL, KLD. But there may still be advantages to using bf16 or f32 for those tensors at long context, but I am not really sure and it would require some other kinds of benchmarks.

I'd have to see your full commands, but I don't see any reason why AesSedai's quant should be 3-4 times slower at all.

model=/ubergarm/Qwen3.5-35B-A3B-GGUF/Qwen3.5-35B-A3B-IQ4_KS-q8_0.gguf
CUDA_VISIBLE_DEVICES="0," \
./build/bin/llama-sweep-bench \
  --model "$model" \
  -c 135168 \
  -ngl 999 \
  -ub 4096 -b 4096 \
  --merge-qkv \
  -muge \
  --threads 1 \
  --no-mmap \
  -n 128 \
  --warmup-batch

model=/ubergarm/Qwen3.5-35B-A3B-GGUF/Qwen3.5-35B-A3B-IQ4_KS-bf16.gguf
CUDA_VISIBLE_DEVICES="0," \
./build/bin/llama-sweep-bench \
  --model "$model" \
  -c 135168 \
  -ngl 999 \
  -ub 4096 -b 4096 \
  --merge-qkv \
  -muge \
  --threads 1 \
  --no-mmap \
  -n 128 \
  --warmup-batch

model=/ubergarm/Qwen3.5-35B-A3B-GGUF/Qwen3.5-35B-A3B-IQ4_KS-f32.gguf
CUDA_VISIBLE_DEVICES="0," \
./build/bin/llama-sweep-bench \
  --model "$model" \
  -c 135168 \
  -ngl 999 \
  -ub 4096 -b 4096 \
  --merge-qkv \
  -muge \
  --threads 1 \
  --no-mmap \
  -n 128 \
  --warmup-batch

What backend are you running this on?

His own, obviously. (I noticed “HugstonOne Enterprise Edition 1.0.9” being mentioned and felt compelled to investigate.😀)

@Maxxim69

interesting haha... i see in the bottom of that github README it mentions that it bundles llama.cpp, but i didn't check to see if there is a pin'd version via submodule etc...

so its llama.cpp under the hood :gucci:

Wow I missed a lot I see, (I thought I closed the discussion and got caught with work, so...).
Thanks to everyone for the reply.

@AesSedai >I thought their non-UD quants were the regular llama.cpp variety but they've changed it up quietly. CSI at work LOL, you keep nailing it.

What´s puzzling me is that Autoround (intel) which do kind of the same thing, runs fine on par with llama.cpp if converted and quantized, keeping the 8bpw quality. I haven´t tested yet which of you have higher precision but both your produced quants are better than average.

@Ubergarm , @Maxxim69 , Basically I am running on llama.cpp, tried both Hugstonized backend or simply the last available build of llama.cpp. It should be slower if we using 8bpw in q5 right, do you guys get same inference speed, I am confused.

Here the prompt: C:---------------------------hugstonone\resources\app.asar.unpacked\runtimes\gpu\hugston-cli.exe --model C:----------------------------\qwen3.5\tested-AesSedaiQwen3.5-35B-A3B-GGUF\qwen3.5 35b q5\Qwen3.5-35B-A3B-Q5_K_M-00001-of-00002.gguf --threads 28 --ctx-size 64000 --n-gpu-layers -1 --batch-size 512 --n-predict -1 --no-warmup --cache-type-k q8_0
ggml_cuda_init: found x CUDA devices:
It is easy for me to test TG always so here the 35b q5_k_m same prompt, query, parameters, output les than 100 tokens : AesSedai quant results [ Prompt: 19.6 t/s | Generation: 24.7 t/s ]
Unsloth q5_k-m [ Prompt: 23.0 t/s | Generation: 25.5 t/s ]
Then we test the heavy weights, Qwen3.5 397b q4_k_m . Here we need more output because it is still loading while printing output. Once fully loaded/offloaded the results:
AesSedai quants 244gb Q4K-M

So what am I doing wrong? How do you load it?

So what am I doing wrong? How do you load it?

I have an example build and run command for ik_llama.cpp here: https://huggingface.co/ubergarm/Qwen3.5-35B-A3B-GGUF#quick-start

If you have 2+ GPUs on ik_llama.cpp you can use -sm graph.

On mainline drop --merge-qkv -muge and use one of AesSedai's new pre-fused GGUFs.

Cheers!

On mainline drop --merge-qkv -muge and use one of AesSedai's new pre-fused GGUFs.
I will try asap :)

Cheers!

@Trilogix1 turns out the issue IS the fused up + gate. I re-quanted the Q5_K_M with a fused and unfused base and when the model is entirely on VRAM you get identical TG and a 10% PP boost. But as soon as you have the model on VRAM + RAM, the TG takes a huge dip. I'll be updating my Qwen3.5 repos to include both fused and unfused quants over the next couple of days, so people can pick which one works for their setup.

I filed a ticket about this issue and am17an provided a patch that looks like it fixes it: https://github.com/ggml-org/llama.cpp/issues/20883#issuecomment-4109411761

So there should be a PR coming from him in the near future and that should bring the fused mixed-offloading TG performance back up.

@AesSedai amazing work, this will change everything.
Seems is done: "am17an closed this as completedin #209106 hours ago"

I have to try it, if this works will beat Intel Autoround on speed keeping the high quality.
That means we can now run proprietary quality model weights in consumer hardware with a decent speed.
Did you receive on offer yet from OpenAi or Anthropic? :))) You will be off market soon I guess.
I wonder if you have the time to create a repo with a good readme for the entire method!
Many thanks.
Edit: Tried it and works, with the last build it got 5.1tps according to my counter or to the llama.cpp counter: [ Prompt: 3.2 t/s | Generation: 4.7 t/s ], x3 times inference speed improvement, no need to update the weights :