Downtown-Case/OpenBuddy_CoTGen-32B-V1-exl3-4.3bpw-hb8

EXL3

Custom exl3 quantization, with 5bpw attention layers, 4bpw for the MLP layers, and an 8bpw lm_head.

Fits ~64K context in 24GB quite well.

Original Model Card:

Prompt format

<|im_start|>system
You are a helpful assistant for generating thinking process.
You are generating thinking process based on the question and existing answer.<|im_end|>
<|im_start|>user
{question}<|im_end|>
<|im_start|>assistant
{answer}

<think>