Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,75 @@
+---
+base_model: cerebras/Qwen3-Coder-REAP-25B-A3B
+tags:
+- awq
+- 4-bit
+- moe
+- reap
+- rdna4
+- gfx1201
+- rocm
+- sglang
+- code
+- thinking
+- quantized
+license: apache-2.0
+---
+
+# Qwen3-Coder-REAP-25B-A3B AWQ 4-bit
+
+AWQ 4-bit quantization of [Cerebras Qwen3-Coder-REAP-25B-A3B](https://huggingface.co/cerebras/Qwen3-Coder-REAP-25B-A3B) — a REAP-pruned ([arxiv:2510.13999](https://arxiv.org/abs/2510.13999)) variant of `Qwen3-Coder-30B-A3B-Instruct` — calibrated with thinking + code data, optimized for AMD RDNA4 (gfx1201) inference with [SGLang](https://github.com/sgl-project/sglang).
+
+## Model Details
+
+|  |  |
+|---|---|
+| **Base model** | [cerebras/Qwen3-Coder-REAP-25B-A3B](https://huggingface.co/cerebras/Qwen3-Coder-REAP-25B-A3B) (REAP prune of [Qwen3-Coder-30B-A3B-Instruct](https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct)) |
+| **Architecture** | Qwen3 MoE (96 experts post-REAP, top-8) |
+| **Parameters** | ~25B total / 3B active |
+| **Pruning method** | REAP (Router-aware Expert pruning, 25% drop) — distinct from REAM (expert merging) |
+| **Layers** | 48 |
+| **Context** | 131K (tested), 256K supported by base |
+| **Quantization** | Native AWQ 4-bit, group_size=128, fused Triton GEMM |
+| **Calibration** | GPTQ via llmcompressor, 256 samples × 1024 tokens, `code_thinking` mix (AM-Thinking-v1, NuminaMath-CoT, ultrachat); ignore=`lm_head, mlp.gate, shared_expert.*` |
+
+## Performance (2x AMD Radeon AI PRO R9700, TP=2, fp8 KV)
+
+`sglang.bench_serving`, single user, FP8 KV cache, `--disable-cuda-graph`:
+
+| Context | TPOT (ms) | tok/s |
+|--------:|----------:|------:|
+|     128 |      43.6 |  22.9 |
+|    1024 |      43.7 |  22.9 |
+|    8192 |      44.1 |  22.7 |
+|   32768 |      44.2 |  22.6 |
+|   65536 |      45.5 |  22.0 |
+|  131072 |      45.6 |  21.9 |
+
+Flat ~22.5 tok/s decode across the full 131K range — A3B MoE stays bandwidth-bound, no attention scaling cliff.
+
+## Notes
+
+This is **REAP**, not REAM. REAP prunes experts based on router-aware impact scores; REAM (Samsung SAIL) instead merges similar experts. Both shrink MoE models, but with different algorithms and tradeoffs — they're not interchangeable. The base Cerebras prune drops 32 of 128 experts (25% reduction).
+
+The CT (compressed-tensors) format from llmcompressor was converted to native AWQ via [`convert_moe_ct_to_awq.py`](https://github.com/mattbucci/2x-R9700-RDNA4-GFX1201-sglang-inference/blob/main/scripts/quantize/convert_moe_ct_to_awq.py) — on ROCm the AWQ Triton GEMM kernel is 6× faster than the compressed-tensors path on the same weights.
+
+`shared_expert.{gate,up,down}_proj` and `mlp.gate` (router) are preserved in BF16 to avoid the always-on residual / routing path going through INT4. `shared_expert_gate` (output dim 1) auto-falls-back to BF16 in the converter since AWQ packing requires divisibility by 8.
+
+## Usage with SGLang
+
+Tested on the [RDNA4 inference stack](https://github.com/mattbucci/2x-R9700-RDNA4-GFX1201-sglang-inference) (SGLang v0.5.10 + 16 RDNA4 patches):
+
+```bash
+git clone https://github.com/mattbucci/2x-R9700-RDNA4-GFX1201-sglang-inference
+cd 2x-R9700-RDNA4-GFX1201-sglang-inference
+./scripts/setup.sh
+MODEL=mattbucci/Qwen3-Coder-REAP-25B-A3B-AWQ scripts/launch.sh coder-reap-25b
+```
+
+The `coder-reap-25b` preset auto-detects the AWQ format and uses `--quantization moe_wna16` with FP8 KV cache for 131K context single-user.
+
+For other inference engines, this is a standard AWQ 4-bit checkpoint (group_size=128, asymmetric, fused MoE) and should load via `vllm` / `transformers + autoawq` without modification.
+
+## Hardware
+
+Calibrated and benchmarked on 2× AMD Radeon AI PRO R9700 (gfx1201, RDNA4, 64 GB total VRAM) with ROCm 7.2 and SGLang v0.5.10 + RDNA4 patches. Per-GPU: ~6 GB weights + ~4 GB FP8 KV at 131K + overhead.