| --- |
| license: apache-2.0 |
| library_name: mlx |
| tags: |
| - mlx |
| - apple-silicon |
| - speculative-decoding |
| - qwen3 |
| - dflash |
| - offline-bundle |
| base_model: |
| - mlx-community/Qwen3-4B-bf16 |
| - z-lab/Qwen3-4B-DFlash-b16 |
| --- |
| |
| # micromodel-ship |
|
|
| Offline Apple Silicon inference bundle for Qwen3-4B with DFlash exact speculative decoding. |
|
|
| **Source code:** [github.com/florianleibert/micromodels](https://github.com/florianleibert/micromodels) |
|
|
| This repo hosts the shippable offline tarball (`micromodel-ship-offline.tar.gz`) that is too large to live in the GitHub repo. The tarball contains the full runnable runtime, both model payloads (target + draft), and the helper scripts needed to serve a local OpenAI-compatible API. |
|
|
| ## Contents |
|
|
| `micromodel-ship-offline.tar.gz` bundles: |
|
|
| - target model: `mlx-community/Qwen3-4B-bf16` |
| - DFlash draft: `z-lab/Qwen3-4B-DFlash-b16` |
| - MLX-based runtime with exact speculative decoding |
| - minimal OpenAI-compatible API server (`POST /v1/chat/completions`) |
| - run, chat, serve, and benchmark scripts |
|
|
| ## Quick start |
|
|
| ```bash |
| curl -L -o micromodel-ship-offline.tar.gz \ |
| https://huggingface.co/florianleibert/micromodel-ship/resolve/main/micromodel-ship-offline.tar.gz |
| tar -xzf micromodel-ship-offline.tar.gz |
| cd micromodel-ship |
| uv sync |
| ./scripts/serve.sh |
| ``` |
|
|
| Health check: |
|
|
| ```bash |
| curl http://127.0.0.1:8051/healthz |
| ``` |
|
|
| ## Performance |
|
|
| Measured on Apple M5 Max, macOS 26.4, parallel-replay verifier, 101-token prompt: |
|
|
| | Max new tokens | Runtime | Generation tok/s | End-to-end tok/s | |
| |---:|---|---:|---:| |
| | 512 | Plain MLX-LM BF16 | 55.13 | 48.67 | |
| | 512 | DFlash BF16 | 190.73 | 186.89 | |
| | 1024 | Plain MLX-LM BF16 | 48.18 | 44.05 | |
| | 1024 | DFlash BF16 | 159.35 | 157.98 | |
|
|
| Observed: `3.46x` decode / `3.84x` end-to-end speedup at 512 tokens. Full numbers in the GitHub repo's `PERFORMANCE.md`. |
|
|
| ## Requirements |
|
|
| - Apple Silicon (M-series) |
| - macOS |
| - Python with `uv` |
|
|
| ## Links |
|
|
| - GitHub repo (source, issues, releases): [florianleibert/micromodels](https://github.com/florianleibert/micromodels) |
|
|
