Mixture of Experts (MoE)
Collection
Sometimes I finetune models specifically to take on expert roles in a MoE configuration, sometimes I find interesting models others have fine tuned. • 8 items • Updated
theprint-MoE-8x3-0126-GGUF
An 18B parameter Mixture of Experts model combining 8 specialized 3B experts, with 2 experts activated per token by default (configurable up to 4 at inference).
Architecture
- Base model: theprint/GeneralChat-Llama3.2-3B (provides shared attention layers)
- Total parameters: ~18B
- Active parameters: ~5B (2 experts) or ~9B (4 experts)
- Gate mode: Hidden (prompt-based router initialization)
Full Model
For more information about this model, including access to the safetensor files, please see theprint/theprint-moe-8x3-0126.
- Downloads last month
- 2,073
Hardware compatibility
Log In to add your hardware
2-bit
3-bit
4-bit
5-bit
6-bit
8-bit
16-bit
Model tree for theprint/theprint-moe-8x3-0126-GGUF
Base model
theprint/theprint-moe-8x3-0126