Instructions to use mradermacher/Mixture-Code-Qwen2.5-Coder-3B-i1-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mradermacher/Mixture-Code-Qwen2.5-Coder-3B-i1-GGUF with Transformers:

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("mradermacher/Mixture-Code-Qwen2.5-Coder-3B-i1-GGUF", dtype="auto")

llama-cpp-python

How to use mradermacher/Mixture-Code-Qwen2.5-Coder-3B-i1-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="mradermacher/Mixture-Code-Qwen2.5-Coder-3B-i1-GGUF",
	filename="Mixture-Code-Qwen2.5-Coder-3B.i1-IQ1_M.gguf",
)

llm.create_chat_completion(
	messages = "No input example has been defined for this model task."
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use mradermacher/Mixture-Code-Qwen2.5-Coder-3B-i1-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf mradermacher/Mixture-Code-Qwen2.5-Coder-3B-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf mradermacher/Mixture-Code-Qwen2.5-Coder-3B-i1-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf mradermacher/Mixture-Code-Qwen2.5-Coder-3B-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf mradermacher/Mixture-Code-Qwen2.5-Coder-3B-i1-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf mradermacher/Mixture-Code-Qwen2.5-Coder-3B-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf mradermacher/Mixture-Code-Qwen2.5-Coder-3B-i1-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf mradermacher/Mixture-Code-Qwen2.5-Coder-3B-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf mradermacher/Mixture-Code-Qwen2.5-Coder-3B-i1-GGUF:Q4_K_M

Use Docker

docker model run hf.co/mradermacher/Mixture-Code-Qwen2.5-Coder-3B-i1-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use mradermacher/Mixture-Code-Qwen2.5-Coder-3B-i1-GGUF with Ollama:
```
ollama run hf.co/mradermacher/Mixture-Code-Qwen2.5-Coder-3B-i1-GGUF:Q4_K_M
```

Unsloth Studio new

How to use mradermacher/Mixture-Code-Qwen2.5-Coder-3B-i1-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for mradermacher/Mixture-Code-Qwen2.5-Coder-3B-i1-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for mradermacher/Mixture-Code-Qwen2.5-Coder-3B-i1-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for mradermacher/Mixture-Code-Qwen2.5-Coder-3B-i1-GGUF to start chatting

Pi new

How to use mradermacher/Mixture-Code-Qwen2.5-Coder-3B-i1-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf mradermacher/Mixture-Code-Qwen2.5-Coder-3B-i1-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "mradermacher/Mixture-Code-Qwen2.5-Coder-3B-i1-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use mradermacher/Mixture-Code-Qwen2.5-Coder-3B-i1-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf mradermacher/Mixture-Code-Qwen2.5-Coder-3B-i1-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default mradermacher/Mixture-Code-Qwen2.5-Coder-3B-i1-GGUF:Q4_K_M

Run Hermes

hermes

Docker Model Runner
How to use mradermacher/Mixture-Code-Qwen2.5-Coder-3B-i1-GGUF with Docker Model Runner:
```
docker model run hf.co/mradermacher/Mixture-Code-Qwen2.5-Coder-3B-i1-GGUF:Q4_K_M
```

Lemonade

How to use mradermacher/Mixture-Code-Qwen2.5-Coder-3B-i1-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull mradermacher/Mixture-Code-Qwen2.5-Coder-3B-i1-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Mixture-Code-Qwen2.5-Coder-3B-i1-GGUF-Q4_K_M

List all available models

lemonade list

mradermacher commited on 26 days ago

Commit

9936402

verified ·

1 Parent(s): 0a8eacc

uploaded from leia

Browse files

Files changed (25) hide show

.gitattributes +24 -0
Mixture-Code-Qwen2.5-Coder-3B.i1-IQ1_M.gguf +3 -0
Mixture-Code-Qwen2.5-Coder-3B.i1-IQ1_S.gguf +3 -0
Mixture-Code-Qwen2.5-Coder-3B.i1-IQ2_M.gguf +3 -0
Mixture-Code-Qwen2.5-Coder-3B.i1-IQ2_S.gguf +3 -0
Mixture-Code-Qwen2.5-Coder-3B.i1-IQ2_XS.gguf +3 -0
Mixture-Code-Qwen2.5-Coder-3B.i1-IQ2_XXS.gguf +3 -0
Mixture-Code-Qwen2.5-Coder-3B.i1-IQ3_M.gguf +3 -0
Mixture-Code-Qwen2.5-Coder-3B.i1-IQ3_S.gguf +3 -0
Mixture-Code-Qwen2.5-Coder-3B.i1-IQ3_XS.gguf +3 -0
Mixture-Code-Qwen2.5-Coder-3B.i1-IQ3_XXS.gguf +3 -0
Mixture-Code-Qwen2.5-Coder-3B.i1-IQ4_NL.gguf +3 -0
Mixture-Code-Qwen2.5-Coder-3B.i1-IQ4_XS.gguf +3 -0
Mixture-Code-Qwen2.5-Coder-3B.i1-Q2_K.gguf +3 -0
Mixture-Code-Qwen2.5-Coder-3B.i1-Q2_K_S.gguf +3 -0
Mixture-Code-Qwen2.5-Coder-3B.i1-Q3_K_L.gguf +3 -0
Mixture-Code-Qwen2.5-Coder-3B.i1-Q3_K_M.gguf +3 -0
Mixture-Code-Qwen2.5-Coder-3B.i1-Q3_K_S.gguf +3 -0
Mixture-Code-Qwen2.5-Coder-3B.i1-Q4_0.gguf +3 -0
Mixture-Code-Qwen2.5-Coder-3B.i1-Q4_1.gguf +3 -0
Mixture-Code-Qwen2.5-Coder-3B.i1-Q4_K_M.gguf +3 -0
Mixture-Code-Qwen2.5-Coder-3B.i1-Q4_K_S.gguf +3 -0
Mixture-Code-Qwen2.5-Coder-3B.i1-Q5_K_M.gguf +3 -0
Mixture-Code-Qwen2.5-Coder-3B.i1-Q5_K_S.gguf +3 -0
Mixture-Code-Qwen2.5-Coder-3B.i1-Q6_K.gguf +3 -0

.gitattributes CHANGED Viewed

@@ -34,3 +34,27 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 Mixture-Code-Qwen2.5-Coder-3B.imatrix.gguf filter=lfs diff=lfs merge=lfs -text

 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 Mixture-Code-Qwen2.5-Coder-3B.imatrix.gguf filter=lfs diff=lfs merge=lfs -text
+Mixture-Code-Qwen2.5-Coder-3B.i1-IQ1_M.gguf filter=lfs diff=lfs merge=lfs -text
+Mixture-Code-Qwen2.5-Coder-3B.i1-IQ1_S.gguf filter=lfs diff=lfs merge=lfs -text
+Mixture-Code-Qwen2.5-Coder-3B.i1-IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text
+Mixture-Code-Qwen2.5-Coder-3B.i1-IQ2_S.gguf filter=lfs diff=lfs merge=lfs -text
+Mixture-Code-Qwen2.5-Coder-3B.i1-IQ2_XS.gguf filter=lfs diff=lfs merge=lfs -text
+Mixture-Code-Qwen2.5-Coder-3B.i1-IQ2_XXS.gguf filter=lfs diff=lfs merge=lfs -text
+Mixture-Code-Qwen2.5-Coder-3B.i1-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
+Mixture-Code-Qwen2.5-Coder-3B.i1-IQ3_S.gguf filter=lfs diff=lfs merge=lfs -text
+Mixture-Code-Qwen2.5-Coder-3B.i1-IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text
+Mixture-Code-Qwen2.5-Coder-3B.i1-IQ3_XXS.gguf filter=lfs diff=lfs merge=lfs -text
+Mixture-Code-Qwen2.5-Coder-3B.i1-IQ4_NL.gguf filter=lfs diff=lfs merge=lfs -text
+Mixture-Code-Qwen2.5-Coder-3B.i1-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
+Mixture-Code-Qwen2.5-Coder-3B.i1-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
+Mixture-Code-Qwen2.5-Coder-3B.i1-Q2_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+Mixture-Code-Qwen2.5-Coder-3B.i1-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
+Mixture-Code-Qwen2.5-Coder-3B.i1-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Mixture-Code-Qwen2.5-Coder-3B.i1-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+Mixture-Code-Qwen2.5-Coder-3B.i1-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
+Mixture-Code-Qwen2.5-Coder-3B.i1-Q4_1.gguf filter=lfs diff=lfs merge=lfs -text
+Mixture-Code-Qwen2.5-Coder-3B.i1-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Mixture-Code-Qwen2.5-Coder-3B.i1-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+Mixture-Code-Qwen2.5-Coder-3B.i1-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Mixture-Code-Qwen2.5-Coder-3B.i1-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+Mixture-Code-Qwen2.5-Coder-3B.i1-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text

Mixture-Code-Qwen2.5-Coder-3B.i1-IQ1_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b9a6146c4a34f8a7c2a3a113ea206f34fdbdc3cbda2c977d4b2c1aeb666ffb7f
+size 850027968

Mixture-Code-Qwen2.5-Coder-3B.i1-IQ1_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:fc2ebbdaa56b6ff2f4791df9e256a6ab65648098dcad212a4423c1eb4587e14c
+size 791094720

Mixture-Code-Qwen2.5-Coder-3B.i1-IQ2_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:06501ea93fe09646789d67111841127599180592d7f86540f3ec0565ca45cf05
+size 1140516288

Mixture-Code-Qwen2.5-Coder-3B.i1-IQ2_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:91739dfcc0ba55d276de0cc694ceefef2ac11970528d71e2d57c6e5ecd9fa16f
+size 1061938624

Mixture-Code-Qwen2.5-Coder-3B.i1-IQ2_XS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c1b241162a6716935c61bed0149fc11e59f51f0e7b493adc9511e54445a35ed9
+size 1031546304

Mixture-Code-Qwen2.5-Coder-3B.i1-IQ2_XXS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4f5d7f36e7783fd4152c97ad81ab4e9a41261c6916b7a2d0f55c4f50b968aa05
+size 948250048

Mixture-Code-Qwen2.5-Coder-3B.i1-IQ3_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:09413ae3bb2f07b7b5625bb1e9f2871361465ed899e666b48c020b78a794f8f6
+size 1488895424

Mixture-Code-Qwen2.5-Coder-3B.i1-IQ3_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f9aab9fed1ecf853401e8ce1044ab81e5c3a4e00a21effd6813896e90014c32e
+size 1456864704

Mixture-Code-Qwen2.5-Coder-3B.i1-IQ3_XS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0a0264a37585ef7eb0f30d0636400c0daf9fe0a3c53a3ffc9547f979dc02bd0d
+size 1391836608

Mixture-Code-Qwen2.5-Coder-3B.i1-IQ3_XXS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:505f6e45c06e00f64318a4ae0c85aac7e15c64b364ddd611ef5bb825d9f3fcf6
+size 1282827712

Mixture-Code-Qwen2.5-Coder-3B.i1-IQ4_NL.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:815c19ba296c0ee746f4170e403aec3e72a35bea30fc2c67633019169ea190a7
+size 1825209792

Mixture-Code-Qwen2.5-Coder-3B.i1-IQ4_XS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4365800658d0fb45c99b6f43c5a0de7bc174a5b1fca78d052473434790e632af
+size 1739095488

Mixture-Code-Qwen2.5-Coder-3B.i1-Q2_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b535b985021621b2d036cb86d045eaaeac2b27cd7b79f63e9852b9a91a2c4afd
+size 1274756544

Mixture-Code-Qwen2.5-Coder-3B.i1-Q2_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ecef3414e2557a30f372f4ede76b50002f23f292f5136f7097dd85c7c5c3f498
+size 1198128576

Mixture-Code-Qwen2.5-Coder-3B.i1-Q3_K_L.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9e98d79d9e3e79c957a1a04a689b236d678dbdf815ceaa2cb9717eeff98ad2e8
+size 1707392448

Mixture-Code-Qwen2.5-Coder-3B.i1-Q3_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:08fa009a606a2c8838931055ced0fbc72e71fbf48c6873fe8c67b94fe0ef1ddb
+size 1590476224

Mixture-Code-Qwen2.5-Coder-3B.i1-Q3_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f02690cb9c6089f7839b94e234196d01fd8043b3ec97f7c0398e120c181e95dd
+size 1454357952

Mixture-Code-Qwen2.5-Coder-3B.i1-Q4_0.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2e4c81336e5f84c915909be775712128c7907ec887c15da8814c76e7a2c8f174
+size 1828486592

Mixture-Code-Qwen2.5-Coder-3B.i1-Q4_1.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4bb8c8b20f8aece476a3cbae0fb76faafae8224dc8656398c247dbf6f906db10
+size 1996258752

Mixture-Code-Qwen2.5-Coder-3B.i1-Q4_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:76d4526bd12d6db76f97d271fb1ed362d9810f31f2bf363953e4792dd4213388
+size 1929903552

Mixture-Code-Qwen2.5-Coder-3B.i1-Q4_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a8dc672b93cbcfefaf621af6dff8f56774f340523ba95ccf26b8be946ece66cc
+size 1834384832

Mixture-Code-Qwen2.5-Coder-3B.i1-Q5_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a409d9122582eee99c591d28b28358dc5667d7f19718c2820e3d4ee39ce744c8
+size 2224815552

Mixture-Code-Qwen2.5-Coder-3B.i1-Q5_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d06221660bd30d9c5ca29de21dd03d86d7abee9bf83e9a2e8735288eb6bf01ec
+size 2169667008

Mixture-Code-Qwen2.5-Coder-3B.i1-Q6_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:67c442888a0d9334096b8f524ebe51e6bc54cae3f6b2669dca31a117efae935c
+size 2538159552