Ornstein-Hermes-3.6-27B-SABER-4bit-MTPLX-Optimized-Speed

MLX 4-bit build of GestaltLabs/Ornstein-Hermes-3.6-27b-SABER packaged for fast local serving with lightning-mlx.

The checkpoint includes an MTPLX sidecar (mtp.safetensors) and runtime metadata (mtplx_runtime.json) so lightning-mlx can use its Qwen3.5 MTPLX serving path on Apple Silicon. Runtime metadata verified on Darwin arm64 with mtplx_version: 0.1.0rc3, mtp_depth_max: 1, recommended_profile: sustained.

The model is the SABER-ablated variant of Ornstein-Hermes-3.6-27B (Qwen3.5, 27B dense, multimodal vision+text). Refer to the source model card for capabilities, license, and SABER details.

Note on MTP weights: mtp.safetensors is packed from the upstream Qwen/Qwen3.6-27B MTP module. The base model itself is the Ornstein-Hermes SABER fine-tune; speculative decoding acceptance rate may differ from upstream.

MTP layout: dense MLP (not MoE). Differs from the 35B-A3B MTPLX bundles, which carry expert-shaped MTP tensors. The dense layout is supported by lightning-mlx convert-mtplx (no expert/router/shared_expert tensors).

Install lightning-mlx

python3 -m pip install git+https://github.com/samuelfaj/lightning-mlx.git

Or:

curl -fsSL https://raw.githubusercontent.com/samuelfaj/lightning-mlx/main/install.sh | bash

Verify:

lightning-mlx --help

Serve this model

From Hugging Face:

lightning-mlx serve samuelfaj/Ornstein-Hermes-3.6-27B-SABER-4bit-MTPLX-Optimized-Speed

From a local checkout:

lightning-mlx serve /path/to/Ornstein-Hermes-3.6-27B-SABER-4bit-MTPLX-Optimized-Speed

Daemon mode:

lightning-mlx serve samuelfaj/Ornstein-Hermes-3.6-27B-SABER-4bit-MTPLX-Optimized-Speed --daemon
lightning-mlx status
lightning-mlx tui <PID-or-model-name>
lightning-mlx kill <PID-or-model-name>

OpenAI-compatible API

curl http://localhost:8010/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{{
    "model": "local",
    "messages": [
      {{"role": "user", "content": "Write a tiny Python HTTP server."}}
    ],
    "stream": true
  }}'

Why use lightning-mlx

lightning-mlx is built for local agent workloads on Apple Silicon: short streamed turns, tool calls, growing context, repeated low-latency interactions. With this checkpoint it uses the packaged MTPLX metadata and Qwen3.5 serving preset instead of treating the model as a generic MLX checkpoint.

The runtime focuses on:

• OpenAI-compatible local serving
• Fast streamed chat completions
• Qwen3.5 reasoning and tool-use paths
• MTPLX-style speculative decoding support
• Daemon, status, TUI, and kill controls

Convert similar local MTPLX models

lightning-mlx convert-mtplx \
  /path/to/Model-MLX-quantized \
  --mtp-source /path/to/Model-with-mtp-tensors

Output is written next to the source as <source>-MTPLX-Optimized-Speed. Then:

lightning-mlx serve /path/to/Model-MLX-quantized-MTPLX-Optimized-Speed

Use with mlx-vlm

This checkpoint is a Qwen3.5 vision-language model. Use mlx-vlm for direct generate / chat without lightning-mlx:

pip install -U mlx-vlm
python -m mlx_vlm.generate \
  --model samuelfaj/Ornstein-Hermes-3.6-27B-SABER-4bit-MTPLX-Optimized-Speed \
  --prompt "Describe this image." \
  --image /path/to/image.jpg \
  --max-tokens 200

Intended use

Research and red-teaming. SABER reduces broad over-refusal while measuring behavioral drift from the base model. Deploy behind your own policy/logging layer.

License

other — see the base model for terms.