Your quants are fantastic - great in both tk/s (speed) and intelligence. The speed is so good, that I would trade part of it for even better precision (quality) in the intelligence. Kindly asking for Q5-6 quants.
Oh, and any plans for dense Qwen3.6-27B? It would be much appreciated 🙏
Thank you for the kind words! Really glad to hear you’re finding the quants useful.
On larger quants, our experiments show that the 4-bit model is already almost on par with BF16, so bigger quants did not show a compelling speed/size vs. quality trade-off in our tests. That said, if you have a specific task, benchmark, or set of prompts where these models are lagging behind higher quants, we’d be very happy to investigate. That kind of feedback would actually be very helpful for improving our quants.
For the 27B model, with 3.7 coming soon, it is not very likely that we will release it. It takes us some time to properly evaluate a good set of models, and 27B dense is also much slower than the MoEs.
Is this the absolute best 4bit quant of qwen3.6 35b ? Cause in all the quants I saw for this model, you need q5-q6 to be able to say it's 99.9 the same as q8, which is pretty much same as f16. What is the raw data you have behind the claim that q4 is almost on par with bf16 ?
For our model releases, we optimize across three objectives: speed, size, and quality. The goal is not to focus on only one of them, but to find the best overall trade-off: strong quality, high speed, and the smallest possible model size.
Our quality score is measured across six benchmarks, including:
BFCL-V3
LiveCodeBench V6
HumanEval
GSM8K
IFEVAL
GSM8K_V (evaluated in both thinking and instruct modes)
So when you ask whether this is the "absolute best" 4-bit quant, the answer depends on what you mean by best. If you mean the highest absolute benchmark score, then no, there are models that score slightly higher. For example, in our benchmarks:
Mudler-APEX-Quality scores 99.5 on average
Our GPU-5 scores 99.27
But our GPU-5 is also about 25% smaller and runs more than 10% faster. That means it can fit into smaller VRAM budgets while delivering almost the same measured quality. For us, that is the trade-off we are trying to optimize.
Another useful comparison is Unsloth's UD-IQ4_XS. It is very close to our GPU-5 model, about 1.5% smaller and 0.2% higher in score, but in our tests it runs even slower than Mudler-APEX-Quality. So again, the question becomes whether that small quality difference is worth the speed cost.
We are always working to improve both our methods and our evaluation process. That includes ShapeLearn, which learns the best datatype for each tensor, as well as how we benchmark, compare, and validate model quality. For this Qwen 3.6 series, we also found that sampling parameters matter quite a bit. With the right settings, we think the improvements are noticeable in practice.
That said, we are planning a blog post soon that will go deeper into model quality, benchmarking, KLD, and the data behind these comparisons.
We rely heavily on community feedback to improve, and we really appreciate everyone's comments, questions, and participation. :)
