Trinity-Mini-oQ
Collection
3 items • Updated
Trinity-Mini-oQ4
oQ4 mixed-precision MLX quantization produced via oMLX.
- Quantization: oQ4 (sensitivity-driven mixed precision, group_size=64)
- Format: MLX safetensors
- Compatible with: mlx-lm, mlx-vlm, oMLX on Apple Silicon
Usage
from mlx_lm import load, generate
model, tokenizer = load("bearzi/Trinity-Mini-oQ4")
prompt = tokenizer.apply_chat_template(
[{"role": "user", "content": "Hello"}],
add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=512, verbose=True))
About oQ
oQ measures per-layer quantization sensitivity through calibration and allocates bits where they matter most — critical layers stay at higher precision, tolerant layers compress aggressively. Target averages of 2/3/4/6/8 bits are provided; actual per-layer bits vary by measured sensitivity.
See oQ documentation.
Comparative benchmarks and feedback welcome — please open a discussion.
- Downloads last month
- 26
Model size
4B params
Tensor type
BF16
·
U32 ·
Hardware compatibility
Log In to add your hardware
4-bit
Model tree for bearzi/Trinity-Mini-oQ4
Base model
arcee-ai/Trinity-Mini-Base-Pre-Anneal Finetuned
arcee-ai/Trinity-Mini-Base Finetuned
arcee-ai/Trinity-Mini
# Make sure mlx-lm is installed # pip install --upgrade mlx-lm # Generate text with mlx-lm from mlx_lm import load, generate model, tokenizer = load("bearzi/Trinity-Mini-oQ4") prompt = "Write a story about Einstein" messages = [{"role": "user", "content": prompt}] prompt = tokenizer.apply_chat_template( messages, add_generation_prompt=True ) text = generate(model, tokenizer, prompt=prompt, verbose=True)