apertus-8b-soc-gguf

GGUF conversion of Colby/apertus-8b-soc, a LoRA fine-tune of swiss-ai/Apertus-8B-2509 on marcodsn/SOC-2508 (Synthetic Online Conversations).

Quantizations

File	Format	Size
apertus-8b-soc-f16.gguf	FP16	~16 GB
apertus-8b-soc-q8_0.gguf	Q8_0	~8 GB
apertus-8b-soc-q5_k_m.gguf	Q5_K_M	~5 GB
apertus-8b-soc-q4_k_m.gguf	Q4_K_M	~4 GB

Ollama usage

hf download Colby/apertus-8b-soc-gguf apertus-8b-soc-q4_k_m.gguf
ollama create apertus-soc:8b -f Modelfile   # FROM ./apertus-8b-soc-q4_k_m.gguf
ollama run apertus-soc:8b