README.md

---
license: apache-2.0
base_model: Qwen/Qwen3-8B
tags:
  - quantized
  - 4-bit
  - int4
  - qwen3
language:
  - en
pipeline_tag: text-generation
---

# Qwen3-8B-AWQ-INT4

INT4 quantization of [`Qwen/Qwen3-8B`](https://huggingface.co/Qwen/Qwen3-8B). Built to run on a single 12 GB+ consumer GPU.

## Footprint

| | |
|---|---|
| Source params | 8B |
| Quantized weights | ~5.7 GB on disk |
| Inference VRAM (incl. KV cache @ 32K context) | ~10 GB |

Fits any 12 GB+ consumer card: RTX 3060 / 4060 / 4070 / 5070, even some integrated mobile GPUs with shared memory. No homelab needed.

## Bench

Scored on [`drawais/needle-1M-bench-mvp`](https://huggingface.co/datasets/drawais/needle-1M-bench-mvp) (50K-token haystack, real arxiv text):

| Metric | Score |
|---|---|
| Overall recall | **80.0%** |
| Paper-anchored | 80.0% |
| Synthetic codes | 80.0% |

## Quick start

```bash
vllm serve drawais/Qwen3-8B-AWQ-INT4 --quantization awq_marlin --max-model-len 32768
```

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
tok = AutoTokenizer.from_pretrained("drawais/Qwen3-8B-AWQ-INT4")
model = AutoModelForCausalLM.from_pretrained("drawais/Qwen3-8B-AWQ-INT4", device_map="auto")
```

## Context length

Native: 40,960 tokens. For longer contexts, enable YaRN rope-scaling per the base model's config.

## License

Apache 2.0 (inherits from base model).