ebircak/gemma-4-31B-it-4bit-W4A16-GPTQ

Gemma-4-31B-it 4-bit GPTQ Quantization (W4A16)

This is a 4-bit GPTQ quantization of Google's Gemma-4-31B-it instruction-tuned multimodal model, optimized for deployment on NVIDIA GPUs with vLLM.

Model Details

Property	Value
Base Model	google/gemma-4-31B-it
Quantization Method	GPTQ W4A16 (asymmetric)
Weight Precision	4-bit int4
Activation Precision	FP16/BF16
Group Size	128
Quantization Library	llm-compressor 0.10.0.1
Format	compressed-tensors (pack-quantized)
Architecture	Gemma4ForConditionalGeneration
Layers	60 decoder layers
Hidden Size	5376
Context Window	262K tokens
Vision Tower	SigLIP (27 layers, NOT quantized)
Quantized Components	Text decoder + projector (vision tower preserved in BF16)

Hardware Requirements

• Verified Deployment: NVIDIA RTX PRO 6000 (96GB VRAM, Blackwell sm120)
• Actual VRAM Usage: ~35GB with gpu_memory_utilization: 0.4 (full 256K context)
• CUDA Version: cu130 (CUDA 13.0)
• vLLM Version: 0.18.2+ (vllm/vllm-openai:gemma4-cu130 Docker image)

Note: The 4-bit GPTQ quantization significantly reduces VRAM requirements compared to BF16 (~62GB). The vision tower remains in BF16, but the quantized text decoder enables deployment on high-end GPUs with substantial headroom for KV cache.

Quantization Recipe

This model was quantized using the following configuration:

default_stage:
  default_modifiers:
    GPTQModifier:
      targets: [Linear]
      ignore:
        - lm_head
        - model.vision_tower.*
        - model.multi_modal_projector.*
      scheme: W4A16_ASYM
      block_size: 128
      dampening_frac: 0.01
      actorder: static
      offload_hessians: false

Calibration Dataset: Mixed dataset including:

• ise-uiuc/Magicoder-Evol-Instruct-110K (1024 samples)
• Salesforce/APIGen-MT-5k (512 samples)
• NousResearch/hermes-function-calling-v1 (512 samples)

Total: 2048 samples at 2048 tokens each (~4M tokens calibration set)

Usage

Loading with Transformers

from transformers import AutoModelForMultimodalLM, AutoTokenizer

model_id = "ebircak/gemma-4-31B-it-4bit-W4A16-GPTQ"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype="auto"
)

Loading with vLLM

Example vLLM Configuration (adapted from actual deployment):

# vllm_config.yaml
model: /path/to/ebircak/gemma-4-31B-it-4bit-W4A16-GPTQ
quantization: compressed-tensors  # GPTQ W4A16
kv_cache_dtype: fp8_e4m3          # FP8 KV cache
gpu_memory_utilization: 0.4       # ~35GB VRAM usage on RTX PRO 6000; can be increased up to 0.95 if needed
max_model_len: 262144             # 256K context
tensor_parallel_size: 1           # Single GPU
enable_prefix_caching: true
enable_chunked_prefill: true

# Gemma4-specific (REQUIRED for tool calling)
enable_auto_tool_choice: true
tool_call_parser: gemma4          
reasoning_parser: gemma4

Files in This Repository

File	Description
model.safetensors	Quantized model weights (~19.2GB)
config.json	Model configuration with quantization_config
tokenizer.json	Tokenizer vocabulary
tokenizer_config.json	Tokenizer config with chat template
chat_template.jinja	Gemma4 native chat template
generation_config.json	Generation parameters
processor_config.json	Processor configuration
recipe.yaml	Quantization recipe
LICENSE	Apache 2.0 License

License

This quantization is released under the Apache 2.0 License.

The base model google/gemma-4-31B-it is also licensed under Apache 2.0.

See LICENSE for the full text.

Citation

If you use this model in your research, please cite:

@misc{gemma4-31b-gptq-quantization,
  title = {Gemma-4-31B-it 4-bit GPTQ Quantization},
  author = {ebircak},
  year = {2026},
  howpublished = {\url{https://huggingface.co/ebircak/gemma-4-31B-it-4bit-W4A16-GPTQ}},
  note = {Quantized with llm-compressor 0.10.0.1}
}

Disclaimer

This is a community quantization of the Google Gemma-4-31B-it model. While efforts have been made to ensure quality, this model is provided "as is" without warranty of any kind. Users should evaluate the model for their specific use cases.

This quantization would not be possible without the hardware support of Gratex International, a.s. (https://www.gratex.com).

Model Card Version

This model card follows the Model Cards for Model Reporting standard.

---

Original Model: google/gemma-4-31B-it
Quantization Tool: llm-compressor
Quantization Format: compressed-tensors