Quantized version of: Tesslate/UIGEN-T3-32B-Preview
FYI: Standard quants might not perform well https://www.reddit.com/r/LocalLLaMA/comments/1l808xc/comment/mx0yea2/
We found that standard quantization significantly degrades quality and can break the model's reasoning chains. For the best results, we highly recommend running it in BF16 or FP8.
'Make knowledge free for everyone'
Made with
- Downloads last month
- 78
Hardware compatibility
Log In to add your hardware
2-bit
3-bit
4-bit
5-bit
6-bit
8-bit
16-bit