File size: 988 Bytes
dcd22b9 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 | ---
license: apache-2.0
base_model: Qwen/Qwen2.5-Math-7B-Instruct
tags:
- quantized
- 4-bit
- int4
- qwen2.5
- math
language:
- en
pipeline_tag: text-generation
---
# Qwen2.5-Math-7B-Instruct-HQQ-INT4
INT4 quantization of [`Qwen/Qwen2.5-Math-7B-Instruct`](https://huggingface.co/Qwen/Qwen2.5-Math-7B-Instruct). Calibration-free companion to [`drawais/Qwen2.5-Math-7B-Instruct-AWQ-INT4`](https://huggingface.co/drawais/Qwen2.5-Math-7B-Instruct-AWQ-INT4).
## Footprint
| | |
|---|---|
| Source params | 7B (math-specialized) |
| Quantized weights | ~5.3 GB on disk |
| Inference VRAM (incl. KV cache @ 32K context) | ~10 GB |
## Quick start
```python
from transformers import AutoTokenizer
from hqq.models.hf.base import AutoHQQHFModel
tok = AutoTokenizer.from_pretrained("drawais/Qwen2.5-Math-7B-Instruct-HQQ-INT4")
model = AutoHQQHFModel.from_quantized("drawais/Qwen2.5-Math-7B-Instruct-HQQ-INT4", device="cuda")
```
## License
Apache 2.0 (inherits from base model).
|