AdvRahul/Axion-Thinking-21B-A3B-Q4_K_M-GGUF

Axion-Thinking-21B-A3B

Model Highlights

Over the past three months, we have continued to scale the thinking capability of Axion-Thinking-21B-A3B, improving both the quality and depth of reasoning, thereby advancing the competitiveness of lightweight reasoning models in complex reasoning tasks.

Key enhancements include:

• Significantly improved performance on reasoning tasks (logic, math, science, coding, academic benchmarks).
• Efficient tool usage capabilities.
• Enhanced 128K long-context understanding.

This version has extended thinking length. Recommended for highly complex reasoning tasks.

Model Overview

Axion-Thinking-21B-A3B is a text MoE post-trained model, with 21B total parameters and 3B activated per token.

Key	Value
Modality	Text
Training Stage	Posttraining
Params(Total / Activated)	21B / 3B
Layers	28
Heads(Q/KV)	20 / 4
Text Experts(Total / Activated)	64 / 6
Shared Experts	2
Context Length	131072

Quickstart

FastDeploy Inference

python -m fastdeploy.entrypoints.openai.api_server \
       --model AdvRahul/Axion-Thinking-21B-A3B \
       --port 8180 \
       --metrics-port 8181 \
       --engine-worker-queue-port 8182 \
       --load_choices "default_v1" \
       --tensor-parallel-size 1 \
       --max-model-len 131072 \
       --reasoning-parser axion_x1 \
       --tool-call-parser axion_x1 \
       --max-num-seqs 32
vllm serve AdvRahul/Axion-Thinking-21B-A3B

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "AdvRahul/Axion-Thinking-21B-A3B"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
    torch_dtype=torch.bfloat16,
)

prompt = "Give me a short introduction to large language models."
messages = [{"role": "user", "content": prompt}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
model_inputs = tokenizer([text], add_special_tokens=False, return_tensors="pt").to(model.device)

generated_ids = model.generate(**model_inputs, max_new_tokens=1024)
output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist()
generate_text = tokenizer.decode(output_ids, skip_special_tokens=True)
print("generate_text:", generate_text)