Add model card metadata, tags, files table, and quant recipe
Browse filesThis PR expands the model card so it renders properly on the Hub and is more useful to people landing on the page.
Changes:
- **Frontmatter moved to the top of the file** (it's currently at the bottom, so YAML metadata doesn't render). Added:
- `library_name: gguf` (enables the GGUF widget)
- `base_model: deepseek-ai/DeepSeek-V4-Flash` + `base_model_relation: quantized` (links the repo to the upstream)
- `quantized_by: antirez`
- `pipeline_tag: text-generation`
- Discoverability tags (gguf, deepseek-v4-flash, moe, iq2_xxs, q2_k, q4_k, ds4, apple-silicon, metal, …)
- **Original intro preserved verbatim** — the two-sentence description and the link to the ds4 repo are unchanged.
- **Files table** with sizes and a per-file quant summary.
- **Per-tensor-class quant recipe table** (q2 file), derived from the GGUF tensor types — makes the recipe encoded in the filenames explicit and grep-able. Notes that q4 only changes the three routed-expert classes.
- **Short rationale paragraph** for the asymmetric mix.
- **Usage snippet** lifted from the ds4 README.
Happy to adjust phrasing, drop sections, or split into multiple PRs if you prefer.
|
@@ -1,6 +1,83 @@
|
|
| 1 |
-
This quants are specific for the DS4 inference engine. They may work with other inference engines or not (they should, but not the MTP model which requires a specific loader).
|
| 2 |
-
https://github.com/antirez/ds4
|
| 3 |
-
|
| 4 |
---
|
| 5 |
license: mit
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 6 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: mit
|
| 3 |
+
library_name: gguf
|
| 4 |
+
pipeline_tag: text-generation
|
| 5 |
+
base_model: deepseek-ai/DeepSeek-V4-Flash
|
| 6 |
+
base_model_relation: quantized
|
| 7 |
+
quantized_by: antirez
|
| 8 |
+
language:
|
| 9 |
+
- en
|
| 10 |
+
tags:
|
| 11 |
+
- gguf
|
| 12 |
+
- quantized
|
| 13 |
+
- deepseek
|
| 14 |
+
- deepseek-v4
|
| 15 |
+
- deepseek-v4-flash
|
| 16 |
+
- moe
|
| 17 |
+
- mixture-of-experts
|
| 18 |
+
- 2-bit
|
| 19 |
+
- 4-bit
|
| 20 |
+
- iq2_xxs
|
| 21 |
+
- q2_k
|
| 22 |
+
- q4_k
|
| 23 |
+
- ds4
|
| 24 |
+
- apple-silicon
|
| 25 |
+
- metal
|
| 26 |
---
|
| 27 |
+
|
| 28 |
+
# DeepSeek V4 Flash — GGUF for ds4
|
| 29 |
+
|
| 30 |
+
This quants are specific for the DS4 inference engine. They may work with other inference engines or not (they should, but not the MTP model which requires a specific loader).
|
| 31 |
+
|
| 32 |
+
https://github.com/antirez/ds4
|
| 33 |
+
|
| 34 |
+
## Files
|
| 35 |
+
|
| 36 |
+
| File | Size | Routed experts (`ffn_{gate,up,down}_exps`) | Everything else |
|
| 37 |
+
|---|---:|---|---|
|
| 38 |
+
| `DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2.gguf` | 80.8 GiB | `IQ2_XXS` (gate, up) + `Q2_K` (down) | `Q8_0` attn proj / shared experts / output, `F16` router + embed + indexer + compressor + HC, `F32` norms / sinks / bias |
|
| 39 |
+
| `DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2.gguf` | 153.3 GiB | `Q4_K` (all three) | same as above |
|
| 40 |
+
| `DeepSeek-V4-Flash-MTP-Q4K-Q8_0-F32.gguf` | 3.6 GiB | MTP / speculative-decoding support (optional, not standalone). | |
|
| 41 |
+
|
| 42 |
+
Use **q2** on 128 GB Mac machines, **q4** on machines with ≥ 256 GB RAM, pair either with **MTP** for optional speculative decoding.
|
| 43 |
+
|
| 44 |
+
## Quantization recipe
|
| 45 |
+
|
| 46 |
+
The filename is the spec. In detail, for the **q2** file:
|
| 47 |
+
|
| 48 |
+
| Tensor class | Quant | Notes |
|
| 49 |
+
|---|---|---|
|
| 50 |
+
| `blk.*.ffn_gate_exps`, `blk.*.ffn_up_exps` | **`IQ2_XXS`** | routed-expert up/gate |
|
| 51 |
+
| `blk.*.ffn_down_exps` | **`Q2_K`** | routed-expert down (K-quant for quality) |
|
| 52 |
+
| `blk.*.ffn_{gate,up,down}_shexp` | `Q8_0` | shared experts |
|
| 53 |
+
| `blk.*.attn_q_a`, `attn_q_b`, `attn_kv`, `attn_output_a`, `attn_output_b` | `Q8_0` | all attention projections (MLA + low-rank output) |
|
| 54 |
+
| `output.weight` | `Q8_0` | output head |
|
| 55 |
+
| `token_embd.weight` | `F16` | input embedding |
|
| 56 |
+
| `blk.*.ffn_gate_inp` (router) | `F16` | learned router |
|
| 57 |
+
| `blk.*.exp_probs_b` (router bias), `blk.*.attn_sinks`, all `*_norm.weight` | `F32` | |
|
| 58 |
+
| `blk.*.ffn_gate_tid2eid` | `I32` | hash-routing tables (first 3 layers only) |
|
| 59 |
+
| `blk.*.attn_compressor_*`, `blk.*.indexer_*`, `blk.*.hc_*`, `blk.*.output_hc_*` | `F16` / `F32` | DSv4-specific auxiliary blocks |
|
| 60 |
+
|
| 61 |
+
For the **q4** file, only the three routed-expert classes change to `Q4_K`. Everything else is byte-for-byte identical to the q2 recipe.
|
| 62 |
+
|
| 63 |
+
The motivation behind the asymmetry: the routed experts are the majority of the parameter count but each individual expert handles only a fraction of tokens, so aggressive quantization on them costs less in average quality than the same treatment of router, projections, or shared experts. Keeping the decision-making components at `Q8_0` preserves model behavior; crushing the experts buys the size.
|
| 64 |
+
|
| 65 |
+
## Usage
|
| 66 |
+
|
| 67 |
+
```bash
|
| 68 |
+
git clone https://github.com/antirez/ds4
|
| 69 |
+
cd ds4
|
| 70 |
+
./download_model.sh q2 # 128 GB RAM machines
|
| 71 |
+
./download_model.sh q4 # >= 256 GB RAM machines
|
| 72 |
+
./download_model.sh mtp # optional MTP / speculative decoding
|
| 73 |
+
make
|
| 74 |
+
|
| 75 |
+
./ds4 -p "Explain Redis streams in one paragraph."
|
| 76 |
+
./ds4-server --ctx 100000 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192
|
| 77 |
+
```
|
| 78 |
+
|
| 79 |
+
The `download_model.sh` script fetches from this repository, resumes partial downloads, and points `./ds4flash.gguf` at the selected variant.
|
| 80 |
+
|
| 81 |
+
## License
|
| 82 |
+
|
| 83 |
+
MIT. The base model copyright is held by DeepSeek; the GGUFs are redistributed under the base model's release terms.
|