| --- |
| license: other |
| license_name: tencent-hy-community |
| license_link: LICENSE |
| library_name: mlx |
| tags: |
| - mlx |
| - jang |
| - jangtq |
| - jangtq-k |
| - mixed-precision |
| - hy3 |
| - hunyuan |
| - hy_v3 |
| - moe |
| - apple-silicon |
| - 295b |
| - osaurus |
| pipeline_tag: text-generation |
| base_model: tencent/Hy3-preview |
| base_model_relation: quantized |
| --- |
| |
| <p align="center"><img src="osaurus-x-banner.png" width="100%" alt="OsaurusAI"/></p> |
|
|
| # Hy3-preview-JANGTQ_K |
| |
| **Tencent Hy3-preview — 102 GB on disk** (down from ~557 GB BF16 source) — |
| **mixed-bit JANGTQ_K** quantization on routed experts + 8-bit affine |
| elsewhere. ~30 % bigger than `Hy3-preview-JANGTQ` (2-bit on routed |
| experts) in exchange for a measurable quality bump on `down_proj` |
| sensitivity, especially on long-output generation. |
|
|
| - **Source:** [tencent/Hy3-preview](https://huggingface.co/tencent/Hy3-preview) |
| (Hy3 architecture, 295 B total / 21 B active, BF16 native, 256 K |
| context, 80 transformer layers + 1 MTP, 192 routed experts top-8 + 1 |
| shared) |
| - **Quantization:** **mixed-bit MXTQ** on routed experts: |
| - `down_proj`: **4-bit** (4096-out, residual-stream sensitive) |
| - `gate_proj`: **2-bit** (gated by SwiGLU) |
| - `up_proj`: **2-bit** (multiplied with gate) |
| - attention / shared expert / dense layer-0 / embed / lm_head / MTP |
| matmuls: 8-bit affine |
| - RMSNorms / router gate / `expert_bias`: fp16 / fp32 passthrough |
| - **MTP:** layer 80 weights preserved (`mtp_mode=preserved_disabled`); |
| decode is one-token-per-forward until accept/reject speculative loop |
| ships. |
| - **Bundle size:** **102 GB on-disk** across 109 shards |
| - **Runs on:** M4 Max 128 GB / M5 Max 128 GB / Mac Studio 192 GB+ |
|
|
| ## What's in the bundle |
|
|
| | Module | Source dtype | Bundle dtype | |
| |---|---|---| |
| | Routed experts (192 × 3 mats × 79 sparse layers, per-expert layout) | BF16 | **JANGTQ_K**: down 4-bit, gate/up 2-bit | |
| | Attention q/k/v/o + q/k norms | BF16 | 8-bit affine g=64 | |
| | Shared expert (gate/up/down) | BF16 | 8-bit affine g=64 | |
| | Dense layer-0 MLP | BF16 | 8-bit affine g=64 | |
| | `embed_tokens` / `lm_head` | BF16 | 8-bit affine g=64 | |
| | MTP layer matmuls | BF16 | 8-bit affine g=64 (preserved_disabled) | |
| | RMSNorms / `router.gate.weight` / `expert_bias` | BF16 / F32 | fp16 passthrough | |
| |
| `jangtq_runtime.safetensors` sidecar (~22 KB) for Swift runtimes — |
| covers `(in=1536, bits=4)` + `(in=4096, bits=2)` codebooks + sign-flip |
| vectors (Hy3 routed projections have asymmetric `[4096↔1536]` dims). |
| |
| ## Why mixed-bit? |
| |
| Hy3 is top-8 routing, so `JANGTQ` (uniform 2-bit) already averages |
| codebook noise across 8 experts per token and ships coherent. `JANGTQ_K` |
| spends extra bits on `down_proj` — the projection whose output enters |
| the residual stream — to give long-output generation more headroom |
| before residual noise compounds. Same scheme that ZAYA1-8B-JANGTQ_K |
| ships for a strictly harder top-1 routing setup. |
| |
| ## Loading (Python) |
| |
| ```bash |
| pip install jang-tools mlx-lm |
| ``` |
| |
| ```python |
| from jang_tools.load_jangtq import load_jangtq_model |
| |
| model, tokenizer = load_jangtq_model("OsaurusAI/Hy3-preview-JANGTQ_K") |
| |
| chat = tokenizer.apply_chat_template( |
| [{{"role": "user", "content": "What is 2 + 2? Answer briefly."}}], |
| tokenize=False, |
| add_generation_prompt=True, |
| reasoning_effort="no_think", |
| ) |
| ``` |
| |
| `load_jangtq_model` auto-registers `model_type=hy_v3` via |
| `jang_tools.hy3` before building the MLX skeleton. The loader applies |
| the standard SwitchGLU fused gate+up + P15 router compile + P18 QKV |
| fusion patches automatically. |
| |
| ## Reasoning + tools |
| |
| - **Reasoning parser:** `qwen3` (extracts `<think>...</think>` blocks) |
| - **Tool parser:** `hunyuan` (Tencent XML-like: |
| `<tool_calls><tool_call>name<tool_sep><arg_key>k</arg_key><arg_value>v</arg_value></tool_call></tool_calls>`) |
| - **Reasoning effort:** `no_think` (default) | `low` | `high` — pass via |
| `apply_chat_template(..., reasoning_effort="…")` |
| - **Cache:** `kv` (standard GQA cache) |
| |
| ## Runtime support matrix |
| |
| | Surface | Status | |
| |---|---| |
| | `jang-tools` Python (`load_jangtq_model`) | ✅ working — this README's load snippet | |
| | `vmlx-swift-lm` Swift | ✅ working — `Libraries/MLXLLM/Models/Hy3.swift` + JANGTQ dispatch. Same family path that ships ZAYA and Bailing/Ling. | |
| | `vmlx_engine` Python re-export | pending | |
| | MTP speculative decode | preserved-disabled — weights present in bundle, accept/reject loop not yet implemented | |
| |
| ## Credits |
| |
| - **Quantization + MLX runtime:** Jinho Jang (eric@osaurus.ai) |
| - **Source model:** Tencent Hy3-preview team |
| - **License:** [Tencent Hy Community License](LICENSE) — non-commercial, EU/UK/SK |
| excluded; consult the LICENSE for full terms |
|
|
