samuelfaj/distill2-0.6B-4bit-MLX

distill2-0.6B — Expert Language Model for CLI Output (Mini)

distill2-0.6B is the second-generation domain-specific Expert Language Model for CLI output compression and classification. At just 0.6B parameters, it matches or exceeds the accuracy of its 1.7B predecessor while being 2.8x smaller and 20x faster at inference.

Built for the distill engine — an open-source CLI output compression tool.

What is distill?

distill is a tool that takes arbitrary command-line output and reduces it to only what matters. Instead of scrolling through 500 lines of npm install logs, you get:

PASS
24 packages installed, 0 vulnerabilities

Instead of parsing a wall of Terraform plan output, you get:

{"create": 3, "change": 12, "destroy": 0}

distill2-0.6B is the brain behind distill — it understands CLI output and knows what's signal vs noise.

distill2 vs distill1

Trait	distill-1.7B (v1)	distill2-0.6B (v2)
Size	1.7B params	0.6B params (2.8x smaller)
Accuracy	95.0%	98.4%
Inference speed	8.2s/sample	0.47s/sample (17x faster)
Training loss	Val 0.12	Val 0.016
Training	LoRA fp16	QLoRA 4-bit
Memory (train)	29.5 GB	10.7 GB (2.8x lower)

8 Specialized Tasks

Task	What it does	Accuracy
pass_fail	Did the command succeed or fail?	100.0%
safe_review	Is this Terraform plan safe?	100.0%
json_extraction	Pull JSON from noisy logs	100.0%
test_result	Test suite pass/fail?	100.0%
typescript_check	Extract TS compiler errors	100.0%
terraform_plan	Count resources created/changed/destroyed	98.4%
security_audit	Count vulns by severity	96.6%
generic	Free-form summary of any CLI output	93.1%

Performance

Metric	Value
Overall accuracy	98.4%
Tasks at 100%	5 of 8
Tasks ≥95%	7 of 8
Base model	Qwen3-0.6B
Training	QLoRA 4-bit, rank 32, all layers
Iterations	6,000
Dataset	100k synthetic CLI outputs
Training hardware	Apple Silicon (M-series)
Peak GPU memory	10.7 GB

Available Formats

Repo	Format	Size	Platform
distill2-0.6B-4bit-MLX	MLX 4-bit + QLoRA adapter	404 MB	macOS (Apple Silicon)
distill2-0.6B-4bit-GGUF	GGUF Q4_K_M + fp16	378 MB / 1.2 GB	Cross-platform

Usage (MLX)

from mlx_lm import load, generate

# Load base 4-bit model + QLoRA adapter together
model, tokenizer = load(
    "samuelfaj/distill2-0.6B-4bit-MLX",
    adapter_path="samuelfaj/distill2-0.6B-4bit-MLX"
)

messages = [
    {"role": "system", "content": "You compress shell or command output for another model..."},
    {"role": "user", "content": "Command output:\nnpm test\n4 tests passed, 0 failed"}
]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
result = generate(model, tokenizer, prompt=prompt, max_tokens=256)
print(result)

Usage (MLX server)

mlx_lm.server \
  --model samuelfaj/distill2-0.6B-4bit-MLX \
  --adapter-path samuelfaj/distill2-0.6B-4bit-MLX

Why QLoRA?

Unlike post-hoc quantization (which drops JSON extraction accuracy from 100% → 84.8%), QLoRA trains directly on the 4-bit quantized base model. The LoRA adapter learns to compensate for quantization artifacts, preserving full accuracy across all tasks.

Method	Accuracy	JSON	Size
LoRA fp16 + post-hoc 4-bit	97.2%	84.8%	384 MB
QLoRA 4-bit (this model)	98.0%	100%	404 MB

Project

This model powers distill — a CLI output compression engine.

Full Distill Collection