feat: initial GGUF release (F16 + Q8_0/Q5_K_M/Q4_K_M/Q4_K_S)

.gitattributes

@@ -33,3 +33,8 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+gemma-4-26B-A4B-it-assistant.F16.gguf filter=lfs diff=lfs merge=lfs -text
+gemma-4-26B-A4B-it-assistant.Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+gemma-4-26B-A4B-it-assistant.Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+gemma-4-26B-A4B-it-assistant.Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+gemma-4-26B-A4B-it-assistant.Q8_0.gguf filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,121 @@
+---
+license: apache-2.0
+license_link: https://ai.google.dev/gemma/docs/gemma_4_license
+library_name: llama.cpp
+pipeline_tag: text-generation
+base_model: google/gemma-4-26B-A4B-it-assistant
+base_model_relation: quantized
+tags:
+  - gguf
+  - llama.cpp
+  - mtp
+  - multi-token-prediction
+  - speculative-decoding
+  - gemma
+  - gemma-4
+  - atomic-chat
+  - turboquant
+---
+
+# Gemma 4 26B-A4B Assistant — GGUF (Atomic Chat)
+
+GGUF builds of [`google/gemma-4-26B-A4B-it-assistant`](https://huggingface.co/google/gemma-4-26B-A4B-it-assistant) — the official Gemma 4
+**Multi-Token Prediction (MTP)** drafter for
+[`google/gemma-4-26B-A4B-it`](https://huggingface.co/google/gemma-4-26B-A4B-it). Use it as a speculative-decoding
+draft model alongside the matching Gemma 4 target to get a meaningful decoding
+speedup at zero quality loss.
+
+Approximate size: **0.4B (assistant) / 27B MoE target**.
+
+> [!IMPORTANT]
+> These GGUFs use the custom `gemma4_assistant` architecture and **will not
+> load in stock `llama.cpp`**. They require the
+> [`atomic-llama-cpp-turboquant`](https://github.com/AtomicBot-ai/atomic-llama-cpp-turboquant) fork, which adds:
+> - the `gemma4_assistant` MTP drafter arch (incl. the centroid LM head for E2B/E4B),
+> - **TurboQuant** KV-cache quantization (`-ctk turbo3 -ctv turbo3`),
+> - the `--mtp-head` / `--spec-type mtp` runtime flags.
+>
+> Loading these files in upstream `ggml-org/llama.cpp` will fail with an
+> unknown architecture error.
+
+## Files
+
+| File | Quant | Size | Notes |
+|---|---|---:|---|
+| `gemma-4-26B-A4B-it-assistant.F16.gguf` | F16 | 815.6 MB | reference (smallest quality loss vs source) |
+| `gemma-4-26B-A4B-it-assistant.Q8_0.gguf` | Q8_0 | 440.4 MB | near-lossless 8-bit |
+| `gemma-4-26B-A4B-it-assistant.Q5_K_M.gguf` | Q5_K_M | 326.4 MB | balanced k-quant |
+| `gemma-4-26B-A4B-it-assistant.Q4_K_M.gguf` | Q4_K_M | 310.4 MB | recommended default for speculative-decoding draft |
+| `gemma-4-26B-A4B-it-assistant.Q4_K_S.gguf` | Q4_K_S | 306.3 MB | smallest k-quant |
+
+## Quick start
+
+Build the fork:
+
+```bash
+git clone https://github.com/AtomicBot-ai/atomic-llama-cpp-turboquant
+cd atomic-llama-cpp-turboquant
+# Pick one of the platform-specific configurations:
+cmake -B build -DGGML_METAL=ON          # Apple Silicon
+# cmake -B build -DGGML_CUDA=ON         # NVIDIA
+# cmake -B build                        # CPU-only
+cmake --build build --target llama-server llama-cli llama-quantize -j
+```
+
+Download the assistant drafter (this repo) and the matching Gemma 4 target:
+
+```bash
+hf download AtomicChat/gemma-4-26B-A4B-it-assistant-GGUF \
+    --include "*Q4_K_M.gguf" --local-dir ./models
+# Any GGUF build of the matching target model works; e.g. unsloth's:
+hf download unsloth/gemma-4-26B-A4B-it-GGUF \
+    --include "*Q4_K_M*.gguf" --local-dir ./models
+```
+
+Run `llama-server` with MTP speculative decoding + TurboQuant KV cache:
+
+```bash
+./build/bin/llama-server \
+    -m         ./models/gemma-4-26B-A4B-it-Q4_K_M.gguf \
+    --mtp-head ./models/gemma-4-26B-A4B-it-assistant.Q4_K_M.gguf \
+    --spec-type mtp \
+    --draft-block-size 3 --draft-max 8 --draft-min 0 \
+    -ngl 99 -ngld 99 \
+    -ctk turbo3 -ctv turbo3 -ctkd turbo3 -ctvd turbo3 \
+    -fa on -c 16384 --host 127.0.0.1 --port 8080
+```
+
+A ready-made launcher lives at `scripts/run-gemma4-26b-a4b-mtp-server.sh`
+in the fork (`MTP_PRESET=throughput|lift|balanced|quality`).
+
+## How MTP works here
+
+Gemma 4 ships with a small "assistant" head that predicts several future tokens
+from the target model's last hidden state. In `atomic-llama-cpp-turboquant` it
+is loaded as a separate GGUF via `--mtp-head` and drives a custom speculative
+decoder (block_size 3, draft_max 16 for quality preset). The verifier runs the target model in
+parallel, guaranteeing the same output distribution as plain greedy/sampled
+decoding.
+
+## TurboQuant KV cache
+
+`turbo3` is the KV-cache quantization scheme used in this fork; it significantly
+reduces KV memory and bandwidth at long contexts with no measurable quality
+regression on Gemma 4. Apply it to both target and drafter via
+`-ctk turbo3 -ctv turbo3 -ctkd turbo3 -ctvd turbo3`.
+
+## License & attribution
+
+Released under the **Gemma Terms of Use**.
+
+- Original model card: [`google/gemma-4-26B-A4B-it-assistant`](https://huggingface.co/google/gemma-4-26B-A4B-it-assistant)
+- Target model: [`google/gemma-4-26B-A4B-it`](https://huggingface.co/google/gemma-4-26B-A4B-it)
+- License text: <https://ai.google.dev/gemma/docs/gemma_4_license>
+
+## Acknowledgements
+
+- [Google DeepMind](https://deepmind.google/models/gemma/) — Gemma 4 family and the MTP drafters.
+- [`ggml-org/llama.cpp`](https://github.com/ggml-org/llama.cpp) — upstream inference engine.
+- TurboQuant primitives — KV-cache quantization scheme integrated in the fork.
+
+— [Atomic Chat](https://atomic.chat)

gemma-4-26B-A4B-it-assistant.F16.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3dc3f22a85b4e5d13d6382fbf1d965ebf5c521f4fbfb755c17a172798d80159c
+size 855228832

gemma-4-26B-A4B-it-assistant.Q4_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:851300493d74f5648f4475681852e3b238f33e97c4b0187734a37b4195b488f9
+size 325452192

gemma-4-26B-A4B-it-assistant.Q4_K_S.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:06426693a8a4171adb945085b3c77d34c98e157dbf02f39e3c7b18639c8093c1
+size 321126816

gemma-4-26B-A4B-it-assistant.Q5_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7273f2b238ee527d88ff75fb3b65d6c8ad6aa0919c7c18363d2d3d219c084124
+size 342262176

gemma-4-26B-A4B-it-assistant.Q8_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0ff5e851eb69aba552efb7cc5da0b37801b42554b403f8584e0b83b92296f69d
+size 461767072