zecanard/Gemopus-4-26B-A4B-it-MLX-5bit-int5-affine

🦆 zecanard/Gemopus-4-26B-A4B-it-MLX-5bit-affine

This model was converted to MLX from Jackrong/Gemopus-4-26B-A4B-it using mlx-vlm version 0.4.4. Please refer to the original model card for more details.

🌟 Quality

Quantized language model with an effective 6.280 bits per weight.

mlx_vlm.convert --quantize --q-group-size 32 --q-bits 5 --q-mode affine

🛠️ Customizations

This quant is aware of the current date, and also enables thinking (if available). You may disable this behavior by deleting the following line from the chat template:

{%- set enable_thinking = true %}

You may also need to adjust your environment’s Reasoning Section Parsing to recognize <|channel>thought as the Start String, and <channel|> as the End String.

🖥️ Use with mlx

pip install -U mlx-vlm

mlx_vlm.generate --model zecanard/Gemopus-4-26B-A4B-it-MLX-5bit-affine --max-tokens 100 --temperature 0 --prompt "Describe this image." --image <path_to_image>