README.md

---
license: apache-2.0
library_name: mlx
base_model:
- Qwen/Qwen3.5-4B
- mlx-community/Qwen3.5-4B-MLX-8bit
pipeline_tag: text-generation
tags:
- mlx
- apple-silicon
- speculative-decoding
- qwen
- qwen3
- qwen3_5
- mtp
- mtplx
- local-ai
- q8
- 8-bit
---

# Qwen3.5-4B Optimized MTPLX (Q8 trunk)

## Run this with MTPLX

**MTPLX** is an MLX-native runtime for native Multi-Token-Prediction speculative decoding on Apple Silicon. Up to **2.24× faster decode** at real coding temperatures (`temp=0.6 / top_p=0.95 / top_k=20`) using the model's own built-in MTP heads — no external drafter, no greedy hack.

```bash
pip install mtplx
mtplx start
```

**Project:** [github.com/youssofal/MTPLX](https://github.com/youssofal/MTPLX)

**Other MTPLX checkpoints:**

- [Qwen3.6-27B-MTPLX-Optimized-Speed](https://huggingface.co/Youssofal/Qwen3.6-27B-MTPLX-Optimized-Speed) — 4-bit flagship speed (63 TPS on M5 Max)
- [Qwen3.6-27B-MTPLX-Optimized](https://huggingface.co/Youssofal/Qwen3.6-27B-MTPLX-Optimized) — verified default (GDN8-Speed4 trunk + CyanKiwi INT4 MTP)
- [Qwen3.5-4B-MTPLX-Optimized-Speed](https://huggingface.co/Youssofal/Qwen3.5-4B-MTPLX-Optimized-Speed) — small 4-bit speed-test

---

Q8-trunk MTPLX speed-test artifact for Apple Silicon.

This model uses the public `mlx-community/Qwen3.5-4B-MLX-8bit` MLX affine 8-bit
trunk and grafts back the official native MTP head from `Qwen/Qwen3.5-4B`. The
MTP head is stored as `mtp.safetensors`; layer-0 attention/MLP linears are
quantized to 4-bit affine group-64, while `mtp.fc` and the MTP norms stay BF16.

## Intended Use

A quick MTPLX download / load / speed-path test artifact at 4B scale, with the
larger 8-bit trunk for higher-fidelity target verification. Once the runtime
ships:

```bash
mtplx start
```

Choose `Custom Hugging Face repo`, then enter:

```text
Youssofal/Qwen3.5-4B-Optimized-MTPLX
```

## Artifact Layout

- Trunk: MLX affine 8-bit, group size 64
- MTP sidecar: official Qwen3.5-4B MTP tensors
- MTP sidecar quantization: body-int4
- Runtime contract: `mtplx_runtime.json`
- MTPLX default: depth 2, target temperature 0.6, draft temperature 0.7

## Local Smoke Result

On the local Apple Silicon MTPLX workstation, the depth-2 speed path measured
**105.21 tok/s** versus **75.63 tok/s** AR on the warm-code prompt
(`max_tokens=48`, `temperature=0.6`, `top_p=0.95`, `top_k=20`).

This Q8 artifact had the best multiplier in the local one-prompt matrix. The
Q4 sibling (`Youssofal/Qwen3.5-4B-MTPLX-Optimized-Speed`) remains faster in
absolute tok/s because its 4-bit trunk has a faster AR baseline.

## Build Stats

```json
{
  "bits": 4,
  "group_size": 64,
  "mode": "affine",
  "output_size_bytes": 86701040,
  "output_tensor_count": 29,
  "policy": "cyankiwi",
  "quantization": "body-int4",
  "quantized_linears": {
    "mtp.layers.0.mlp.down_proj":   {"bits": 4, "group_size": 64, "mode": "affine"},
    "mtp.layers.0.mlp.gate_proj":   {"bits": 4, "group_size": 64, "mode": "affine"},
    "mtp.layers.0.mlp.up_proj":     {"bits": 4, "group_size": 64, "mode": "affine"},
    "mtp.layers.0.self_attn.k_proj":{"bits": 4, "group_size": 64, "mode": "affine"},
    "mtp.layers.0.self_attn.o_proj":{"bits": 4, "group_size": 64, "mode": "affine"},
    "mtp.layers.0.self_attn.q_proj":{"bits": 4, "group_size": 64, "mode": "affine"},
    "mtp.layers.0.self_attn.v_proj":{"bits": 4, "group_size": 64, "mode": "affine"}
  },
  "source_tensor_count": 15
}
```

## Links

- **MTPLX**: [github.com/youssofal/MTPLX](https://github.com/youssofal/MTPLX)  ·  `pip install mtplx`
- **Base model**: [Qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B)
- **Trunk source**: [mlx-community/Qwen3.5-4B-MLX-8bit](https://huggingface.co/mlx-community/Qwen3.5-4B-MLX-8bit)
- **Q4 sibling**: [Youssofal/Qwen3.5-4B-MTPLX-Optimized-Speed](https://huggingface.co/Youssofal/Qwen3.5-4B-MTPLX-Optimized-Speed)