---
license: mit
tags:
  - moe
  - mixture-of-experts
  - hybrid-attention
  - mla
  - lightning-attention
  - mxfp4
  - osaurus
  - mlx
  - bailing
  - ling
  - apple-silicon
base_model: inclusionAI/Ling-2.6-flash
pipeline_tag: text-generation
library_name: mlx
---

<p align="center"><img src="osaurus-x-banner.png" width="100%"/></p>

# Ling-2.6-flash-MXFP4

**~103B-A8B hybrid MoE — 63 GB on disk** (down from the 200 GB bf16 source) —
**stock 4-bit affine** quantization on inclusionAI's Bailing-V2.5 hybrid
architecture. Loads via `mlx_lm.load()` with the `bailing_hybrid` model
class — no TurboQuant runtime, no sidecar required.

- **Source:** [inclusionAI/Ling-2.6-flash](https://huggingface.co/inclusionAI/Ling-2.6-flash)
  (Ant Group's Bailing-V2.5 hybrid: 32 layers MLA + Lightning-Linear-Attention,
  256 experts top-8, MTP head, 131K context)
- **Quantization:** MXFP4 — every weight (routed experts, attention,
  shared experts, dense MLP, embed, lm_head) at **4-bit affine
  group_size=32**. Norms, router gates, expert biases, and slopes stay
  fp16/fp32 passthrough.
- **Bundle size:** **63 GB on-disk** across 51 shards
- **Runs on:** M3 Max 96 GB+ / M4 Max 128 GB / M5 Max 128 GB / Mac Studio

## Why two variants?

| | JANGTQ2 | MXFP4 |
|---|---|---|
| Routed experts | 2-bit MXTQ codebook (Hadamard + Lloyd-Max) | 4-bit affine |
| Attention / shared / dense | 8-bit affine | 4-bit affine |
| Bundle size | 30 GB | 63 GB |
| Quality | tighter (8-bit attention) | uniform 4-bit |
| Loader | `jang_tools.load_jangtq` (TurboQuant kernel) | stock `mlx_lm.load()` |
| Sidecar | required | not needed |
| Min RAM | 64 GB | 96 GB |

JANGTQ2 trades cheap-but-slow MXTQ codec on the routed experts for a tighter
overall bit budget. MXFP4 is the simpler "just-works-with-stock-MLX" option
for users who don't want the TurboQuant runtime in their stack.

## Architecture (`bailing_hybrid`)

Hybrid attention — every 8th layer is full softmax MLA, the other 28 of 32
are Lightning-Linear-Attention. Plus a Multi-Token Prediction head.

| Layer block | Count | Attention | MLP |
|---|---|---|---|
| Layer 0 | 1 | Linear (GLA) | Dense MLP (intermediate=9216) |
| Layers 1–6, 8–14, 16–22, 24–30 | 27 | **Linear (GLA)** | MoE (256+1) |
| Layers 7, 15, 23, 31 | 4 | **MLA** (full softmax) | MoE (256+1) |
| MTP head (32) | 1 | MLA | MoE (256+1) |

See the [JANGTQ variant card](https://huggingface.co/JANGQ-AI/Ling-2.6-flash-JANGTQ)
for the deeper architecture writeup.

## Loading (Python)

```bash
pip install mlx-lm jang-tools
```

```python
from mlx_lm import load, generate
model, tokenizer = load("OsaurusAI/Ling-2.6-flash-MXFP4")
```

Stock `mlx_lm.load()` works once `mlx_lm/models/bailing_hybrid.py` is
present (shipped with `jang-tools >= TBD`). The bundle's
`configuration_bailing_moe_v2_5.py` and `modeling_bailing_moe_v2_5.py`
provide HF compatibility for tooling that goes through transformers.

## Reasoning + tools

Default is **`detailed thinking off`**. To enable:

```python
messages = [
    {"role": "system", "content": "detailed thinking on"},
    {"role": "user",   "content": "..."},
]
```

The model emits `<think>...</think>` reasoning blocks before answers when
thinking is on. DeepSeek-style tool-call format.

## Credits

- **Quantization + MLX runtime:** Jinho Jang ([eric@osaurus.ai](mailto:eric@osaurus.ai))
- **Base model:** [inclusionAI](https://huggingface.co/inclusionAI) — Ant
  Group's Bailing team
- **Architecture references:** Lightning-Attention-2 (arXiv:2401.04658),
  DeepSeek-V2/V3 MLA, MTP from DeepSeek-V3
- **Osaurus:** [osaurus.ai](https://osaurus.ai) — Apple-Silicon-first
  inference for open-weight LLMs.