| --- |
| license: mit |
| license_name: deepseek-license |
| library_name: mlx |
| base_model: deepseek-ai/DeepSeek-V4-Flash |
| base_model_relation: quantized |
| pipeline_tag: text-generation |
| tags: |
| - mlx |
| - jang |
| - jangtq |
| - jangtq2 |
| - jangtq-prestack |
| - mxtq |
| - deepseek |
| - deepseek-v4 |
| - deepseek-v4-flash |
| - moe |
| - mla |
| - hash-layers |
| - mtp |
| - apple-silicon |
| - osaurus |
| --- |
| |
| <p align="center"><img src="osaurus-x-banner.png" width="100%" alt="OsaurusAI"/></p> |
|
|
| # DeepSeek-V4-Flash-JANGTQ2 |
|
|
| **DeepSeek-V4-Flash — 79.6 GB on disk** (down from 149 GB FP4+FP8 source) — |
| uniform **2-bit JANGTQ** quantization on routed experts + 8-bit affine on |
| everything else + preserved MTP head. |
|
|
| - **Source:** [deepseek-ai/DeepSeek-V4-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) |
| (43 transformer layers + 1 MTP head, **256 routed experts top-6 + 1 |
| shared expert**, **3 hash layers**, MLA + mHC residuals, ~284 B total) |
| - **Quantization:** uniform **2-bit MXTQ** on routed-expert MLP + |
| 8-bit affine on attention (`wq_a/wq_b/wkv/wo_a/wo_b`) / shared |
| expert / Compressor / Indexer / embed / lm_head / MTP. RMSNorms, |
| router gate, mHC fn matrices, attn_sink, ape stay fp16/fp32 |
| passthrough. |
| - **Variant:** `std` (preserves MTP layer 43; one-token-per-forward |
| until a JANG runtime ships the accept/reject speculative-decode loop). |
| The companion `DeepSeek-V4-Flash-JANGTQ-K` variant drops MTP for a |
| smaller bundle. |
| - **Routed-expert layout:** **pre-stacked** along axis 0 under |
| `ffn.experts.switch_mlp.{{gate_proj, up_proj, down_proj}}` per the |
| JANGTQ-PRESTACK STANDARD. Sidecar `jangtq_runtime.safetensors` |
| (~24 KB) ships both `(in=2048, bits=2)` and `(in=4096, bits=2)` |
| codebooks + sign-flip vectors for Swift runtimes. |
| - **Bundle size:** **~79.6 GB on-disk** |
| - **Runs on:** M4 Max 128 GB / M5 Max 128 GB / Mac Studio 192 GB+ |
|
|
| ## Why top-6 + 2-bit holds |
|
|
| DSV4-Flash routes through **6 of 256 experts per token** plus 1 always-on |
| shared expert and 3 hash layers — so per-token output averages |
| codebook noise across 7+ pathways. That's a much weaker quality |
| constraint than top-1 architectures (where every token rides a single |
| expert's quant error). MiniMax (top-2) and Hy3-preview (top-8) both |
| ship coherent uniform JANGTQ2; DSV4 sits between them. |
|
|
| ## Loading (Python) |
|
|
| ```bash |
| pip install jang-tools mlx-lm |
| ``` |
|
|
| ```python |
| from jang_tools.load_jangtq import load_jangtq_model |
| |
| model, tokenizer = load_jangtq_model("OsaurusAI/DeepSeek-V4-Flash-JANGTQ2") |
| |
| chat = tokenizer.apply_chat_template( |
| [{{"role": "user", "content": "What is 2 + 2? Answer briefly."}}], |
| tokenize=False, |
| add_generation_prompt=True, |
| ) |
| ``` |
|
|
| `load_jangtq_model` auto-registers `model_type=deepseek_v4` via |
| `jang_tools.dsv4` before building the MLX skeleton. The loader applies |
| the DSV4-specific MLA absorb + fp32 SDPA + mHC + Compressor + Indexer |
| patches automatically. |
|
|
| ## Runtime support matrix |
|
|
| | Surface | Status | |
| |---|---| |
| | `jang-tools` Python (`load_jangtq_model`) | ✅ working | |
| | `vmlx-swift-lm` Swift | ✅ working — `DeepseekV4JANGTQ` family path | |
| | MTP speculative decode | preserved-disabled — weights present (variant=std); accept/reject loop not yet in any JANG runtime | |
|
|
| ## Validated runtime contract |
|
|
| - 43 transformer layers + 1 MTP head materialize; 40 sparse-MoE layers |
| hydrate routed experts via TurboQuantLinear (2-bit MXTQ). |
| - 33,792 MXTQ tensors / 522 affine / 706 passthrough. |
| - Capabilities: `family=deepseek_v4`, `reasoning_parser=deepseek_r1`, |
| `tool_parser=dsml`, `think_in_template=True`, `cache_type=mla`. |
|
|
| ## Reasoning + tools |
|
|
| - **Reasoning parser:** `deepseek_r1` |
| - **Tool parser:** `dsml` (DeepSeek Markup Language — distinct from |
| `deepseek_tool_parser`; see `~/jang/research/DSV4-EVAL-NUANCES.md`) |
| - **Reasoning template:** `<｜thinking_begin｜>...<｜thinking_end｜>` blocks |
| via `enable_thinking=True` (default off — pass-through chat mode). |
| Greedy `T=0` with `enable_thinking=True` collapses into repetition on |
| DSV4; use `T=0.6` for pass@1 like the original DeepSeek release. |
| - **Cache:** `mla` (Multi-head Latent Attention with kv_lora_rank=512) |
|
|
| ## Credits |
|
|
| - **Quantization + MLX runtime:** Jinho Jang (eric@osaurus.ai) |
| - **Source model:** DeepSeek AI |
| - **License:** MIT, inherited from upstream |
|
|
