---
license: mit
library_name: transformers
tags:
- deepseek-v4
- mixture-of-experts
- moe
- mhc
- csa
- hca
- scaffold
- random-init
pipeline_tag: text-generation
---

# DeepSeek-V4 Mini (3B) — randomly-initialized architecture replica

Faithful small-scale (~3.2B total / ~1.10B activated per token)
replica of the DeepSeek-V4 architecture, sized to be trainable on rented GPUs
and to map cleanly onto the full-scale V4-Flash dimensions for weight slicing.

This is a **randomly-initialized** scaffold — generates noise. Its purpose:
- reference architecture for ablation / hyperparameter-search experiments
- target for weight transfer / slicing from real V4-Pro / V4-Flash

## Architecture summary

| | Value |
|---|---|
| hidden_size | 1536 |
| num_hidden_layers | 28 |
| num_attention_heads | 24 |
| num_key_value_heads | 1 (MQA) |
| head_dim | 64 |
| q_lora_rank / o_lora_rank | 512 / 512 |
| qk_rope_head_dim | 32 |
| o_groups | 4 |
| n_routed_experts | 24 |
| n_shared_experts | 1 |
| num_experts_per_tok | 4 |
| num_hash_layers | 2 |
| moe_intermediate_size | 768 |
| compress_ratios | [0, 0, 4, 112, 4, 112, 4, 112, 4, 112, 4, 112, 4, 112, 4, 112, 4, 112, 4, 112, 4, 112, 4, 112, 4, 112, 4, 0] |
| index_topk / heads / head_dim | 192 / 16 / 96 |
| sliding_window | 64 |
| max_position_embeddings | 1,048,576 (YaRN factor=16) |
| vocab_size | 129280 (real V4-Flash tokenizer) |
| num_nextn_predict_layers | 1 (V3-style MTP) |
| hc_mult (n_hc) | 4 |
| Storage dtype | bfloat16 |

## Quick start

```python
from huggingface_hub import login, snapshot_download
login()  # private repo

local = snapshot_download(repo_id="kshitijthakkar/deepseek-v4-mini-3B-init")

import sys, os
sys.path.insert(0, os.path.join(local, "code"))
import deepseek_v4  # registers DeepseekV4{Config,ForCausalLM} with HF auto classes

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained(local)
model = AutoModelForCausalLM.from_pretrained(local, torch_dtype=torch.bfloat16)
model.eval()

ids = tok.apply_chat_template(
    [{"role": "user", "content": "Hello"}],
    return_tensors="pt", add_generation_prompt=True, return_dict=True,
)
with torch.no_grad():
    out = model(input_ids=ids["input_ids"])
print(out.logits.shape)
```

## Components implemented

mHC (Sinkhorn-Knopp) · CSA + Lightning Indexer · HCA · pure sliding-window ·
Shared-KV MQA + grouped output projection (per-group `wo_a`) · partial RoPE +
output `-i` rotation · attention sink · DeepseekMoE with `sqrt(softplus)`
routing · hash-routed early layers · clamped SwiGLU · MTP head · YaRN.

Every component is bit-equivalent in math to the official `inference/model.py`
+ `kernel.py:hc_split_sinkhorn` (FP4/FP8 quantization and Hadamard rotation
are skipped — those are inference optimizations, not architecture).

## Citation

```bibtex
@misc{deepseek_v4_2026,
  author = {DeepSeek-AI},
  title  = {DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence},
  year   = {2026},
  url    = {https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash}
}
```