README.md

---
license: apache-2.0
language:
- en
tags:
- mlx
- distill
- distill-mini
- cli
- code
- compression
- qwen
- qwen3
- expert-model
- domain-specific
- task-specialized
- lora
- qlora
pipeline_tag: text-generation
base_model: Qwen/Qwen3-0.6B
---

# distill2-0.6B — Expert Language Model for CLI Output (Mini)

**distill2-0.6B** is the second-generation **domain-specific Expert Language Model** for CLI output compression and classification. At just 0.6B parameters, it matches or exceeds the accuracy of its 1.7B predecessor while being 2.8x smaller and 20x faster at inference.

Built for the **[distill](https://github.com/samuelfaj/distill)** engine — an open-source CLI output compression tool.

## What is distill?

[distill](https://github.com/samuelfaj/distill) is a tool that takes arbitrary command-line output and reduces it to only what matters. Instead of scrolling through 500 lines of `npm install` logs, you get:

```
PASS
24 packages installed, 0 vulnerabilities
```

Instead of parsing a wall of Terraform plan output, you get:

```json
{"create": 3, "change": 12, "destroy": 0}
```

**distill2-0.6B is the brain behind distill** — it understands CLI output and knows what's signal vs noise.

## distill2 vs distill1

| Trait | distill-1.7B (v1) | distill2-0.6B (v2) |
|-------|-------------------|---------------------|
| **Size** | 1.7B params | 0.6B params (2.8x smaller) |
| **Accuracy** | 95.0% | **98.4%** |
| **Inference speed** | 8.2s/sample | **0.47s/sample** (17x faster) |
| **Training loss** | Val 0.12 | **Val 0.016** |
| **Training** | LoRA fp16 | **QLoRA 4-bit** |
| **Memory (train)** | 29.5 GB | 10.7 GB (2.8x lower) |

## 8 Specialized Tasks

| Task | What it does | Accuracy |
|------|-------------|----------|
| `pass_fail` | Did the command succeed or fail? | 100.0% |
| `safe_review` | Is this Terraform plan safe? | 100.0% |
| `json_extraction` | Pull JSON from noisy logs | 100.0% |
| `test_result` | Test suite pass/fail? | 100.0% |
| `typescript_check` | Extract TS compiler errors | 100.0% |
| `terraform_plan` | Count resources created/changed/destroyed | 98.4% |
| `security_audit` | Count vulns by severity | 96.6% |
| `generic` | Free-form summary of any CLI output | 93.1% |

## Performance

| Metric | Value |
|--------|-------|
| Overall accuracy | **98.4%** |
| Tasks at 100% | 5 of 8 |
| Tasks ≥95% | 7 of 8 |
| Base model | Qwen3-0.6B |
| Training | QLoRA 4-bit, rank 32, all layers |
| Iterations | 6,000 |
| Dataset | 100k synthetic CLI outputs |
| Training hardware | Apple Silicon (M-series) |
| Peak GPU memory | 10.7 GB |

## Available Formats

| Repo | Format | Size | Platform |
|------|--------|------|----------|
| **distill2-0.6B-4bit-MLX** | MLX 4-bit + QLoRA adapter | 404 MB | macOS (Apple Silicon) |
| [distill2-0.6B-4bit-GGUF](https://huggingface.co/samuelfaj/distill2-0.6B-4bit-GGUF) | GGUF Q4_K_M + fp16 | 378 MB / 1.2 GB | Cross-platform |

## Usage (MLX)

```python
from mlx_lm import load, generate

# Load base 4-bit model + QLoRA adapter together
model, tokenizer = load(
    "samuelfaj/distill2-0.6B-4bit-MLX",
    adapter_path="samuelfaj/distill2-0.6B-4bit-MLX"
)

messages = [
    {"role": "system", "content": "You compress shell or command output for another model..."},
    {"role": "user", "content": "Command output:\nnpm test\n4 tests passed, 0 failed"}
]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
result = generate(model, tokenizer, prompt=prompt, max_tokens=256)
print(result)
```

## Usage (MLX server)

```bash
mlx_lm.server \
  --model samuelfaj/distill2-0.6B-4bit-MLX \
  --adapter-path samuelfaj/distill2-0.6B-4bit-MLX
```

## Why QLoRA?

Unlike post-hoc quantization (which drops JSON extraction accuracy from 100% → 84.8%), QLoRA trains directly on the 4-bit quantized base model. The LoRA adapter learns to compensate for quantization artifacts, preserving full accuracy across all tasks.

| Method | Accuracy | JSON | Size |
|--------|----------|------|------|
| LoRA fp16 + post-hoc 4-bit | 97.2% | 84.8% | 384 MB |
| **QLoRA 4-bit (this model)** | **98.0%** | **100%** | **404 MB** |

## Project

This model powers [distill](https://github.com/samuelfaj/distill) — a CLI output compression engine.

[Full Distill Collection](https://huggingface.co/collections/samuelfaj/distill-6a0606f9b131c289025659fc)