Carnice-27b-MLX-oQ8

This repo is a straight MLX oQ8 quant of kai-os/Carnice-27b for local Apple Silicon inference.

No other edits, additions, merges, or behavioral changes have been made to the model beyond the quantization/export step.

M1 Ultra Mac Studio Throughput

Measured on a Mac Studio with Apple M1 Ultra and 128 GB unified memory.

• Carnice-27b full weights: 10.982 tokens/sec average generation, 53.984 GB peak memory
• Carnice-27b-MLX-oQ8: 25.002 tokens/sec average generation, 18.123 GB peak memory

oQ8 Quant

This is a mixed 4/8 MLX quant.

• less important weights are kept at 4-bit
• more important weights are kept at 8-bit
• final exported model size works out to about 5.31 bits per weight

Original Model

The text below is carried over from kai-os/Carnice-27b, with the quant-specific notes above added for this MLX release.

Carnice-27b is the merged full-model release of the Trinity Hermes-Agent training run on top of Qwen/Qwen3.5-27B.

This repo contains the quantized MLX export of that model.

Acknowledgements

This work would not have been possible without Zachary Mueller, Lambda, Teknium, and Nous Research.

Trained using traces from lambda/hermes-agent-reasoning-traces

Trinity Process

Stage A: Premium Reasoning Backbone

• 3300 train rows
• 193 validation rows
• 12288 max length
• final eval loss 0.5316
• final eval perplexity 1.7016

Stage B: Hermes Alignment

• widened Carnice + DJ + Lambda alignment mix
• 2269 train rows
• 80 validation rows
• final eval loss 0.2336
• final eval perplexity 1.2632

Stage C: Carnice Polish

• 600 train rows
• 60 validation rows
• final eval loss 0.2310
• final eval perplexity 1.2599

Intended Use

Carnice-27b is tuned for Hermes-Agent style terminal, file, browser, repo, debugging, and multi-step tool workflows.

Benchmark Status

Reproducible benchmark runs are not attached yet. They will be added only after the dedicated benchmark box run is complete.

Loading with mlx-lm

python -m mlx_lm.generate \
  --model /path/to/Carnice-27b-MLX-oQ8 \
  --prompt "Write a bash command to list large files recursively."