Choose quantization type
#5
by Fileqx - opened
hi! Is it possible to serve unsloth/Qwen3-14B-GGUF via vllm choosing the specific quantized model for example Qwen3-14B-Q4_0.gguf
hi! Is it possible to serve unsloth/Qwen3-14B-GGUF via vllm choosing the specific quantized model for example Qwen3-14B-Q4_0.gguf
Yes pretty sure you can with the dense models