Qwen3.6-27B-MTPLX-Optimized-Speed

Instructions to use finbase0530/Qwen3.6-27B-MTPLX-Optimized-Speed with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use finbase0530/Qwen3.6-27B-MTPLX-Optimized-Speed with MLX:

# Make sure mlx-lm is installed
# pip install --upgrade mlx-lm

# Generate text with mlx-lm
from mlx_lm import load, generate

model, tokenizer = load("finbase0530/Qwen3.6-27B-MTPLX-Optimized-Speed")

prompt = "Write a story about Einstein"
messages = [{"role": "user", "content": prompt}]
prompt = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True
)

text = generate(model, tokenizer, prompt=prompt, verbose=True)

Notebooks
Google Colab
Kaggle
Local Apps
LM Studio

Pi new

How to use finbase0530/Qwen3.6-27B-MTPLX-Optimized-Speed with Pi:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "finbase0530/Qwen3.6-27B-MTPLX-Optimized-Speed"

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "mlx-lm": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "finbase0530/Qwen3.6-27B-MTPLX-Optimized-Speed"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use finbase0530/Qwen3.6-27B-MTPLX-Optimized-Speed with Hermes Agent:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "finbase0530/Qwen3.6-27B-MTPLX-Optimized-Speed"

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default finbase0530/Qwen3.6-27B-MTPLX-Optimized-Speed

Run Hermes

hermes

MLX LM

How to use finbase0530/Qwen3.6-27B-MTPLX-Optimized-Speed with MLX LM:

Generate or start a chat session

# Install MLX LM
uv tool install mlx-lm
# Interactive chat REPL
mlx_lm.chat --model "finbase0530/Qwen3.6-27B-MTPLX-Optimized-Speed"

Run an OpenAI-compatible server

# Install MLX LM
uv tool install mlx-lm
# Start the server
mlx_lm.server --model "finbase0530/Qwen3.6-27B-MTPLX-Optimized-Speed"
# Calling the OpenAI-compatible server with curl
curl -X POST "http://localhost:8000/v1/chat/completions" \
   -H "Content-Type: application/json" \
   --data '{
     "model": "finbase0530/Qwen3.6-27B-MTPLX-Optimized-Speed",
     "messages": [
       {"role": "user", "content": "Hello"}
     ]
   }'

Qwen3.6-27B-MTPLX-Optimized-Speed / README.md

finbase0530

Duplicate from Youssofal/Qwen3.6-27B-MTPLX-Optimized-Speed

0aa3682 18 days ago

preview code

raw

history blame contribute delete

4.51 kB

	---
	library_name: mlx
	license: apache-2.0
	base_model:
	- Qwen/Qwen3.6-27B
	pipeline_tag: text-generation
	tags:
	- mlx
	- apple-silicon
	- speculative-decoding
	- qwen
	- qwen3
	- qwen3-next
	- mtp
	- mtplx
	- local-ai
	---

	# Qwen3.6-27B MTPLX Optimized Speed

	## Run this with MTPLX

	MTPLX is an MLX-native runtime for native Multi-Token-Prediction speculative decoding on Apple Silicon. Up to 2.24× faster decode at real coding temperatures (`temp=0.6 / top_p=0.95 / top_k=20`) using the model's own built-in MTP heads — no external drafter, no greedy hack.

	```bash
	pip install mtplx
	mtplx start
	```

	Project: [github.com/youssofal/MTPLX](https://github.com/youssofal/MTPLX)

	Other MTPLX checkpoints:

	- [Qwen3.6-27B-MTPLX-Optimized](https://huggingface.co/Youssofal/Qwen3.6-27B-MTPLX-Optimized) — verified default (GDN8-Speed4 trunk + CyanKiwi INT4 MTP)
	- [Qwen3.5-4B-MTPLX-Optimized-Speed](https://huggingface.co/Youssofal/Qwen3.5-4B-MTPLX-Optimized-Speed) — small 4-bit speed-test
	- [Qwen3.5-4B-Optimized-MTPLX](https://huggingface.co/Youssofal/Qwen3.5-4B-Optimized-MTPLX) — small 8-bit

	---

	This is the speed checkpoint for MTPLX v0.1.0-preview. It combines the
	MLXCommunity flat 4-bit MLX trunk with the calibrated CyanKiwi INT4 MTP sidecar
	and embeds `mtplx_runtime.json` so MTPLX can run it as the optimized speed
	variant.

	Target repository: `Youssofal/Qwen3.6-27B-MTPLX-Optimized-Speed`

	## Runtime

	```bash
	mtplx pull Youssofal/Qwen3.6-27B-MTPLX-Optimized-Speed
	mtplx run "hello" --model Youssofal/Qwen3.6-27B-MTPLX-Optimized-Speed
	mtplx serve --model Youssofal/Qwen3.6-27B-MTPLX-Optimized-Speed --port 8000 --no-stats-footer
	```

	MTPLX v0.1.0-preview supports Qwen3-Next-MTP only. This model is verified for
	the `qwen3-next-mtp` backend.

	## Sources And Attribution

	\| Component \| Source \| Revision \| License \|
	\|---\|---\|---:\|---\|
	\| Base model \| `Qwen/Qwen3.6-27B` \| `6a9e13bd6fc8f0983b9b99948120bc37f49c13e9` \| Apache-2.0 \|
	\| Calibrated MTP sidecar source \| `cyankiwi/Qwen3.6-27B-AWQ-BF16-INT4` \| `8cc526ddcfdd53346f94fb9b5458ca8495a63218` \| Apache-2.0 \|
	\| MTPLX conversion and runtime contract \| `youssofal/mtplx` \| v0.1.0-preview.1 \| Apache-2.0 \|

	The trunk is the MLXCommunity flat 4-bit affine conversion of
	`Qwen/Qwen3.6-27B`. The MTP sidecar is derived from the downloaded CyanKiwi
	compressed-tensors checkpoint at the revision listed above.

	## Verification

	`mtplx_runtime.json` records:

	- architecture: `qwen3-next-mtp`
	- maximum MTP depth: `3`
	- recommended profile: `performance-cold`
	- recommended draft-only LM head: `3-bit affine, group_size=64`
	- recommended draft sampler: temperature `0.70`, top-p `0.95`, top-k `20`
	- exactness gate: `Phase 0H paged-verifier smoke`
	- exactness max absolute diff: `0.0`
	- verified hardware: `Apple M5 Max, 128 GB unified memory`
	- verified timestamp: `2026-05-03T23:07:00+0100`

	Target sampler used for the recorded contract: temperature `0.6`,
	top-p `0.95`, top-k `20`, with thinking mode disabled. The draft proposal
	sampler is intentionally slightly hotter at temperature `0.70`; target
	verification and residual correction remain exact.

	## Performance Honesty

	This is the default speed lane. On the local Apple M5 Max fanmax
	performance-cold benchmark, this artifact reached 63.056 tok/s and
	62.886 tok/s at depth 3 on the long-code 192-token prompt when using its
	contract-recommended 3-bit draft-only LM head plus draft sampler temperature
	`0.70`, with thinking mode disabled and foreground app load reduced. The
	proposal change reduced the measured correction tokens from 6 to 3 and verify
	calls from 50 to 49 on the recorded prompt; target sampling remains temperature
	`0.6` / top-p `0.95` / top-k `20`. A full greedy diagnostic on the same cleaned
	fanmax window measured 60.108 tok/s. Earlier contract evidence measured
	61.527 and 60.900 tok/s, the earlier 3-bit draft-head contract measured 60.038
	and 60.061 tok/s, and the older 4-bit draft-only LM head measured 57.668 tok/s
	on the same lane.

	## Files

	- `model-*.safetensors`: MLXCommunity flat 4-bit trunk shards.
	- `mtp.safetensors`: calibrated prequantized INT4 MTP sidecar.
	- `mtplx_runtime.json`: MTPLX verified runtime contract.
	- `MTPLX_PUBLISH_MANIFEST.json`: local staging manifest with source paths,
	materialization method, sizes, and optional hashes.

	## Links

	- MTPLX: [github.com/youssofal/MTPLX](https://github.com/youssofal/MTPLX) · `pip install mtplx`
	- Base model: [Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B)