OsaurusAI/Mistral-Medium-3.5-128B-JANGTQ

Quantized mistralai/Mistral-Medium-3.5-128B for Apple Silicon (MLX) — dense 128B parameter multimodal LLM with image input + 256K context.

Source	mistralai/Mistral-Medium-3.5-128B
Architecture	mistral3 wrapper — ministral3 text decoder (88L × 12288, GQA 96/8) + pixtral vision encoder (48L × 1664)
Quant format	JANGTQ (TurboQuant 2-bit + Hadamard, group_size=64)
Bundle size on disk	40.76 GB (37 safetensors shards)
License	Apache-2.0 (inherits from upstream)
Modalities	Text + image in / text out (no audio, no video)
Context	262 144 tokens (YaRN, factor=64 from orig=4096)

What's quantized

• Text decoder linears (88 layers × 7 projections = q/k/v/o + gate/up/down) → TurboQuant 2-bit with Hadamard pre-rotation (group_size=64)
• embed_tokens → affine 8-bit (mx.quantize)
• All RMSNorms (input/post-attention) → fp16 passthrough
• vision_tower.*, multi_modal_projector.*, lm_head → fp16 passthrough (matches upstream modules_to_not_convert)

Vision tower (model.vision_tower.*, 48 layers, 1664 hidden, patch=14, image_size=1540, spatial_merge=2) — kept bf16 → fp16 passthrough, matching upstream quantization_config.modules_to_not_convert. Images dispatch to the same pixtral encoder; embeddings are folded into the LM via a 2-layer GELU multimodal projector (also fp16 passthrough).

lm_head is fp16 passthrough (matches upstream ignored set) — no quantization noise on the final logits.

Codec round-trip validation (this bundle)

7/7 PASS at cosine ≥ 0.94 across L0 → L87 covering attn + MLP + GQA k/v projections — the expected 2-bit TurboQuant noise floor. Source FP8 e4m3 dequant uses weight_scale_inv per-tensor scale; vision/projector/lm_head pass through unchanged.

Run on Apple Silicon

pip install mlx safetensors transformers pillow
python -m jang_tools.mistral3.runtime \
    --src ~/.mlxstudio/models/OsaurusAI/Mistral-Medium-3.5-128B-JANGTQ \
    --prompt "Describe this image." \
    --image /path/to/photo.jpg \
    --max-new 64

The runtime auto-detects weight_format and dispatches; image preprocessing matches the pixtral spec (jang_tools.vl.pixtral.PixtralImageProcessor).

Build

python -m jang_tools.convert_mistral3_jangtq \
    ~/.mlxstudio/models/_sources/Mistral-Medium-3.5-128B \
    ~/.mlxstudio/models/JANGQ-AI/Mistral-Medium-3.5-128B-JANGTQ JANGTQ2

Credits

Quantized by Jinho Jang (eric@osaurus.ai). MLX-native pipeline; 88 dense decoder layers + 48 pixtral vision layers run on M-series Macs.