rungalileo/llama-3.2-3b-instruct-trtllm-ckpt-wq_w8a8sq-kv_int8

Llama-3.2-3B-Instruct TensorRT-LLM checkpoint (W8A8 SmoothQuant + INT8 KV)

TensorRT-LLM checkpoint for Llama-3.2-3B-Instruct, with W8A8 SmoothQuant weight/activation quantization and INT8 KV cache. Use with trtllm-build to produce an engine for inference.

Model details

Item	Value
Base model	Llama-3.2-3B-Instruct
Framework	TensorRT-LLM (checkpoint format)
Weight/activation quantization	W8A8 SmoothQuant (PER_CHANNEL_PER_TOKEN_PLUGIN)
KV cache	INT8
Producer	TensorRT-LLM convert_checkpoint.py with SmoothQuant calibration (--smoothquant 0.5 --per_token --per_channel --int8_kv_cache --calib_size 512)
Architecture	LlamaForCausalLM (decoder-only)

Build (how to produce this checkpoint)

This checkpoint is generated with a direct TensorRT-LLM conversion + SmoothQuant calibration:

python TensorRT-LLM/examples/models/core/llama/convert_checkpoint.py \
  --model_dir /path/to/Llama-3.2-3B-Instruct \
  --output_dir ./llama-3.2-3b-instruct-trtllm-ckpt-wq_w8a8sq-kv_int8 \
  --dtype float16 \
  --tp_size 1 \
  --smoothquant 0.5 \
  --per_token \
  --per_channel \
  --int8_kv_cache \
  --calib_size 512

Output

After conversion, --output_dir contains config.json and rank0.safetensors; that is the checkpoint in this repo.

Upload (how to upload to Hugging Face)

cd ./llama-3.2-3b-instruct-trtllm-ckpt-wq_w8a8sq-kv_int8

huggingface-cli repo create rungalileo/llama-3.2-3b-instruct-trtllm-ckpt-wq_w8a8sq-kv_int8 --repo-type model
huggingface-cli upload rungalileo/llama-3.2-3b-instruct-trtllm-ckpt-wq_w8a8sq-kv_int8 . --repo-type model

How to use

1. Build engine

git clone https://huggingface.co/rungalileo/llama-3.2-3b-instruct-trtllm-ckpt-wq_w8a8sq-kv_int8
cd llama-3.2-3b-instruct-trtllm-ckpt-wq_w8a8sq-kv_int8

trtllm-build --checkpoint_dir . --output_dir ./engine \
  --max_batch_size 1 --max_input_len 512 --max_seq_len 1024

2. Run inference

Use a tokenizer from the base model (for example meta-llama/Llama-3.2-3B-Instruct):

trtllm-serve ./engine --tokenizer meta-llama/Llama-3.2-3B-Instruct --port 8000
# OpenAI-compatible API: http://localhost:8000/v1/completions

Files in this repo

• config.json - TensorRT-LLM model config
• rank0.safetensors - Rank 0 weights (single-GPU)

References

• TensorRT-LLM
• Llama 3.2