mishig HF Staff commited on
Commit
6b25e58
·
verified ·
1 Parent(s): ef3b960

Add model card metadata, tags, files table, and quant recipe

Browse files

This PR expands the model card so it renders properly on the Hub and is more useful to people landing on the page.

Changes:

- **Frontmatter moved to the top of the file** (it's currently at the bottom, so YAML metadata doesn't render). Added:
- `library_name: gguf` (enables the GGUF widget)
- `base_model: deepseek-ai/DeepSeek-V4-Flash` + `base_model_relation: quantized` (links the repo to the upstream)
- `quantized_by: antirez`
- `pipeline_tag: text-generation`
- Discoverability tags (gguf, deepseek-v4-flash, moe, iq2_xxs, q2_k, q4_k, ds4, apple-silicon, metal, …)

- **Original intro preserved verbatim** — the two-sentence description and the link to the ds4 repo are unchanged.

- **Files table** with sizes and a per-file quant summary.

- **Per-tensor-class quant recipe table** (q2 file), derived from the GGUF tensor types — makes the recipe encoded in the filenames explicit and grep-able. Notes that q4 only changes the three routed-expert classes.

- **Short rationale paragraph** for the asymmetric mix.

- **Usage snippet** lifted from the ds4 README.

Happy to adjust phrasing, drop sections, or split into multiple PRs if you prefer.

Files changed (1) hide show
  1. README.md +80 -3
README.md CHANGED
@@ -1,6 +1,83 @@
1
- This quants are specific for the DS4 inference engine. They may work with other inference engines or not (they should, but not the MTP model which requires a specific loader).
2
- https://github.com/antirez/ds4
3
-
4
  ---
5
  license: mit
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: mit
3
+ library_name: gguf
4
+ pipeline_tag: text-generation
5
+ base_model: deepseek-ai/DeepSeek-V4-Flash
6
+ base_model_relation: quantized
7
+ quantized_by: antirez
8
+ language:
9
+ - en
10
+ tags:
11
+ - gguf
12
+ - quantized
13
+ - deepseek
14
+ - deepseek-v4
15
+ - deepseek-v4-flash
16
+ - moe
17
+ - mixture-of-experts
18
+ - 2-bit
19
+ - 4-bit
20
+ - iq2_xxs
21
+ - q2_k
22
+ - q4_k
23
+ - ds4
24
+ - apple-silicon
25
+ - metal
26
  ---
27
+
28
+ # DeepSeek V4 Flash — GGUF for ds4
29
+
30
+ This quants are specific for the DS4 inference engine. They may work with other inference engines or not (they should, but not the MTP model which requires a specific loader).
31
+
32
+ https://github.com/antirez/ds4
33
+
34
+ ## Files
35
+
36
+ | File | Size | Routed experts (`ffn_{gate,up,down}_exps`) | Everything else |
37
+ |---|---:|---|---|
38
+ | `DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2.gguf` | 80.8 GiB | `IQ2_XXS` (gate, up) + `Q2_K` (down) | `Q8_0` attn proj / shared experts / output, `F16` router + embed + indexer + compressor + HC, `F32` norms / sinks / bias |
39
+ | `DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2.gguf` | 153.3 GiB | `Q4_K` (all three) | same as above |
40
+ | `DeepSeek-V4-Flash-MTP-Q4K-Q8_0-F32.gguf` | 3.6 GiB | MTP / speculative-decoding support (optional, not standalone). | |
41
+
42
+ Use **q2** on 128 GB Mac machines, **q4** on machines with ≥ 256 GB RAM, pair either with **MTP** for optional speculative decoding.
43
+
44
+ ## Quantization recipe
45
+
46
+ The filename is the spec. In detail, for the **q2** file:
47
+
48
+ | Tensor class | Quant | Notes |
49
+ |---|---|---|
50
+ | `blk.*.ffn_gate_exps`, `blk.*.ffn_up_exps` | **`IQ2_XXS`** | routed-expert up/gate |
51
+ | `blk.*.ffn_down_exps` | **`Q2_K`** | routed-expert down (K-quant for quality) |
52
+ | `blk.*.ffn_{gate,up,down}_shexp` | `Q8_0` | shared experts |
53
+ | `blk.*.attn_q_a`, `attn_q_b`, `attn_kv`, `attn_output_a`, `attn_output_b` | `Q8_0` | all attention projections (MLA + low-rank output) |
54
+ | `output.weight` | `Q8_0` | output head |
55
+ | `token_embd.weight` | `F16` | input embedding |
56
+ | `blk.*.ffn_gate_inp` (router) | `F16` | learned router |
57
+ | `blk.*.exp_probs_b` (router bias), `blk.*.attn_sinks`, all `*_norm.weight` | `F32` | |
58
+ | `blk.*.ffn_gate_tid2eid` | `I32` | hash-routing tables (first 3 layers only) |
59
+ | `blk.*.attn_compressor_*`, `blk.*.indexer_*`, `blk.*.hc_*`, `blk.*.output_hc_*` | `F16` / `F32` | DSv4-specific auxiliary blocks |
60
+
61
+ For the **q4** file, only the three routed-expert classes change to `Q4_K`. Everything else is byte-for-byte identical to the q2 recipe.
62
+
63
+ The motivation behind the asymmetry: the routed experts are the majority of the parameter count but each individual expert handles only a fraction of tokens, so aggressive quantization on them costs less in average quality than the same treatment of router, projections, or shared experts. Keeping the decision-making components at `Q8_0` preserves model behavior; crushing the experts buys the size.
64
+
65
+ ## Usage
66
+
67
+ ```bash
68
+ git clone https://github.com/antirez/ds4
69
+ cd ds4
70
+ ./download_model.sh q2 # 128 GB RAM machines
71
+ ./download_model.sh q4 # >= 256 GB RAM machines
72
+ ./download_model.sh mtp # optional MTP / speculative decoding
73
+ make
74
+
75
+ ./ds4 -p "Explain Redis streams in one paragraph."
76
+ ./ds4-server --ctx 100000 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192
77
+ ```
78
+
79
+ The `download_model.sh` script fetches from this repository, resumes partial downloads, and points `./ds4flash.gguf` at the selected variant.
80
+
81
+ ## License
82
+
83
+ MIT. The base model copyright is held by DeepSeek; the GGUFs are redistributed under the base model's release terms.