Qwen3-1.7B-w2g64-gptq_v2 (Quantized with BitDistiller)

This model is quantized with BitDistiller (Qingtao Li's fork) from Qwen/Qwen3-1.7B. It is trained on a mixture of wikitext and alpaca via self-distillation.

It is stored in GPTQv2 format ("gptq_v2"), compatible with GPTQModel, vLLM (since v0.11.1), etc.

Benchmark

Below is the 5-shot accuracy of Qwen/Qwen3-1.7B (not quantized), JunHowie/Qwen3-1.7B-GPTQ-Int4 (GPTQ Int4), this model (BitDistiller Int2), and #9bd265d (GPTQv2 Int2), evaluated on 3 benchmarks via lm-eval. This model's output quality is significantly improved compared to #9bd265d, although still having a gap with 4-bit quantization.

Quantization	Bits	ARC Challenge (normalized)	HellaSwag (normalized)	MMLU
None	BF16	52.3%	60.0%	60.2%
GPTQ	Int4	47.9%	56.6%	54.5%
BitDist.	Int2	32.4%	42.1%	38.8%
GPTQv2	Int2	23.1%	32.6%	25.1%

---

The first release (#9bd265d) was quantized with GPTQModel's GPTQv2 quantization, calibrated on a subset of c4.

It was primarily uploaded for testing vLLM's GPTQv2 support.