Instructions to use danielilov/qwen3.5-0.8B-JANGTQ4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use danielilov/qwen3.5-0.8B-JANGTQ4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="danielilov/qwen3.5-0.8B-JANGTQ4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForImageTextToText

processor = AutoProcessor.from_pretrained("danielilov/qwen3.5-0.8B-JANGTQ4")
model = AutoModelForImageTextToText.from_pretrained("danielilov/qwen3.5-0.8B-JANGTQ4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

MLX

How to use danielilov/qwen3.5-0.8B-JANGTQ4 with MLX:

# Make sure mlx-lm is installed
# pip install --upgrade mlx-lm

# Generate text with mlx-lm
from mlx_lm import load, generate

model, tokenizer = load("danielilov/qwen3.5-0.8B-JANGTQ4")

prompt = "Write a story about Einstein"
messages = [{"role": "user", "content": prompt}]
prompt = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True
)

text = generate(model, tokenizer, prompt=prompt, verbose=True)

Notebooks
Google Colab
Kaggle
Local Apps
LM Studio

vLLM

How to use danielilov/qwen3.5-0.8B-JANGTQ4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "danielilov/qwen3.5-0.8B-JANGTQ4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "danielilov/qwen3.5-0.8B-JANGTQ4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/danielilov/qwen3.5-0.8B-JANGTQ4

SGLang

How to use danielilov/qwen3.5-0.8B-JANGTQ4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "danielilov/qwen3.5-0.8B-JANGTQ4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "danielilov/qwen3.5-0.8B-JANGTQ4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "danielilov/qwen3.5-0.8B-JANGTQ4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "danielilov/qwen3.5-0.8B-JANGTQ4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Pi new

How to use danielilov/qwen3.5-0.8B-JANGTQ4 with Pi:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "danielilov/qwen3.5-0.8B-JANGTQ4"

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "mlx-lm": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "danielilov/qwen3.5-0.8B-JANGTQ4"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use danielilov/qwen3.5-0.8B-JANGTQ4 with Hermes Agent:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "danielilov/qwen3.5-0.8B-JANGTQ4"

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default danielilov/qwen3.5-0.8B-JANGTQ4

Run Hermes

hermes

MLX LM

How to use danielilov/qwen3.5-0.8B-JANGTQ4 with MLX LM:

Generate or start a chat session

# Install MLX LM
uv tool install mlx-lm
# Interactive chat REPL
mlx_lm.chat --model "danielilov/qwen3.5-0.8B-JANGTQ4"

Run an OpenAI-compatible server

# Install MLX LM
uv tool install mlx-lm
# Start the server
mlx_lm.server --model "danielilov/qwen3.5-0.8B-JANGTQ4"
# Calling the OpenAI-compatible server with curl
curl -X POST "http://localhost:8000/v1/chat/completions" \
   -H "Content-Type: application/json" \
   --data '{
     "model": "danielilov/qwen3.5-0.8B-JANGTQ4",
     "messages": [
       {"role": "user", "content": "Hello"}
     ]
   }'

Docker Model Runner
How to use danielilov/qwen3.5-0.8B-JANGTQ4 with Docker Model Runner:
```
docker model run hf.co/danielilov/qwen3.5-0.8B-JANGTQ4
```

qwen3.5-0.8B-JANGTQ4

This repository contains a JANGTQ4-converted version of qwen3.5-0.8B intended for local inference with vMLX / MLX-style runtimes.

The model was converted for efficient local text-generation workloads, with the goal of reducing memory usage and improving practical inference performance on Apple Silicon and compatible local setups.

Model details

Base model: qwen3.5-0.8B
Format / variant: JANGTQ4
Primary use: Local text generation and assistant-style inference
Target runtime: vMLX / MLX-compatible local inference stacks
Quantization: TQ4-style JANG conversion

Intended use

This model is intended for local experimentation, development, and inference workflows where a compact Qwen-family model is useful. It may be suitable for:

Lightweight assistant tasks
Local coding-agent experiments
Prompt-format and cache-behavior testing
Low-memory local inference
Fast iteration on Apple Silicon systems

Limitations

This is a converted / quantized model and may differ from the original base model in quality, numerical behavior, formatting behavior, and edge-case reliability. Small models may be more prone to instruction-following mistakes, hallucinations, malformed tool calls, or repetitive output than larger models.

Please test carefully for your own use case.

License

This repository contains a converted model derived from the upstream Qwen model. Use of this model is subject to the license terms of the original base model and any applicable restrictions from the upstream provider. Please review the upstream model license before using or redistributing this conversion.

Downloads last month: 222

Safetensors

Model size

0.3B params

Tensor type

U32

F16

MLX

Hardware compatibility

Quantized