Instructions to use ree2raz/CyberSecQwen-4B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ree2raz/CyberSecQwen-4B-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="ree2raz/CyberSecQwen-4B-GGUF",
	filename="cybersecqwen-4b-Q4_K_M.gguf",
)

llm.create_chat_completion(
	messages = "No input example has been defined for this model task."
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use ree2raz/CyberSecQwen-4B-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf ree2raz/CyberSecQwen-4B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf ree2raz/CyberSecQwen-4B-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf ree2raz/CyberSecQwen-4B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf ree2raz/CyberSecQwen-4B-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf ree2raz/CyberSecQwen-4B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf ree2raz/CyberSecQwen-4B-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf ree2raz/CyberSecQwen-4B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf ree2raz/CyberSecQwen-4B-GGUF:Q4_K_M

Use Docker

docker model run hf.co/ree2raz/CyberSecQwen-4B-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use ree2raz/CyberSecQwen-4B-GGUF with Ollama:
```
ollama run hf.co/ree2raz/CyberSecQwen-4B-GGUF:Q4_K_M
```

Unsloth Studio new

How to use ree2raz/CyberSecQwen-4B-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for ree2raz/CyberSecQwen-4B-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for ree2raz/CyberSecQwen-4B-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for ree2raz/CyberSecQwen-4B-GGUF to start chatting

Docker Model Runner
How to use ree2raz/CyberSecQwen-4B-GGUF with Docker Model Runner:
```
docker model run hf.co/ree2raz/CyberSecQwen-4B-GGUF:Q4_K_M
```

Lemonade

How to use ree2raz/CyberSecQwen-4B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull ree2raz/CyberSecQwen-4B-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.CyberSecQwen-4B-GGUF-Q4_K_M

List all available models

lemonade list

CyberSecQwen-4B-GGUF / README.md

ree2raz

Add CTI-Bench evaluation results (GGUF Q4_K_M)

ac975e9 verified 9 days ago

preview code

raw

history blame contribute delete

3 kB

metadata

license: apache-2.0
base_model: lablab-ai-amd-developer-hackathon/CyberSecQwen-4B
tags:
  - qwen3
  - cybersecurity
  - cti
  - cwe-classification
  - vulnerability-analysis
  - gguf
  - q4_k_m
  - 4-bit
  - quantized

CyberSecQwen-4B-GGUF

GGUF Q4_K_M quantized version of CyberSecQwen-4B.

Quantization

Parameter	Value
Method	GGUF Q4_K_M (llama.cpp)
Weight precision	4-bit (Q4_K_M = 4-bit block-scaled with k-quant importance)
Quantization tool	llama.cpp (build from master)
Conversion tool	convert_hf_to_gguf.py
Quantization hardware	Modal A10G
File	cybersecqwen-4b-Q4_K_M.gguf (2.5 GB)

CTI-Bench Evaluation

Evaluated under the Foundation-Sec-8B protocol:

Temperature 0.3, max_tokens 512, concurrency 8
5 independent trials, zero-shot (no system prompt)
llama.cpp server on Modal L4 GPU

Task	GGUF Q4_K_M	AWQ 4-bit	FP16 Reference
CTI-MCQ (2,500 items)	0.5368 ± 0.0048	0.5921 ± 0.0083	0.5868 ± 0.0029
CTI-RCM (1,000 items)	0.6254 ± 0.0063	0.5814 ± 0.0025	0.6664 ± 0.0023

Key findings:

CTI-RCM (CVE→CWE classification): GGUF Q4_K_M is the best quantized variant (-4.1 pts vs FP16). Superior to AWQ 4-bit by +4.4 points.
CTI-MCQ (CTI knowledge): AWQ 4-bit performs better than GGUF for multiple-choice questions.
GGUF preserves task-specific classification accuracy better due to block-wise k-quant importance scaling.

Trial results

CTI-MCQ

Trial	Seed	Accuracy
1	42	0.5420
2	43	0.5280
3	44	0.5360
4	45	0.5392
5	46	0.5388

CTI-MCQ

Trial	Seed	Accuracy
1	42	0.6270
2	43	0.6300
3	44	0.6270
4	45	0.6300
5	46	0.6130

Quantization variants

Variant	CTI-MCQ	CTI-RCM	Size	Engine
AWQ 4-bit	0.5921	0.5814	2.7 GB	vLLM
GGUF Q4_K_M	0.5368	0.6254	2.5 GB	llama.cpp

Choose GGUF for vulnerability classification, AWQ for MCQ/general chat.

Usage with llama.cpp

# Download
wget https://huggingface.co/ree2raz/CyberSecQwen-4B-GGUF/resolve/main/cybersecqwen-4b-Q4_K_M.gguf

# Serve
./llama-server -m cybersecqwen-4b-Q4_K_M.gguf --host 0.0.0.0 --port 8080 -ngl 99 -c 4096

Model Size

Format	Size
Original FP16	~8 GB
GGUF Q4_K_M	~2.5 GB

Citation

@misc{{cybersecqwen2026,
  title  = {{CyberSecQwen-4B: A Compact CTI Specialist}},
  author = {{Mulia, Samuel}},
  year   = {{2026}},
  url    = {{https://huggingface.co/athena129/CyberSecQwen-4B}}
}}

Evaluation Infrastructure

GitHub repository — Modal scripts for quantization + evaluation.