Quantized Models (GGUF, IQ, Imatrix)
Collection
Various GGUF quantizations of small models. Models with a "checkmark" are personal favorites. An "orange arrow" means it's being uploaded. • 97 items • Updated • 71
Updated!
Please grab "v2" quants remade with the new tokenizer settings to fix the endless generation issues.
SillyTavern
The complete AIO recommended preset:
v2-SillyTavern-Presets-AIO-2024-12-28.json
My GGUF-ARM-Imatrix quants of Captain-Eris_Twighlight-V0.420-12B.
⛶ [Expand/hide] Example setup.
Example setup in SillyTavern...
- Downloads last month
- 122
Hardware compatibility
Log In to add your hardware
3-bit
4-bit
5-bit
6-bit
8-bit
16-bit
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support