Instructions to use RedHatAI/Qwen3.6-27B-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use RedHatAI/Qwen3.6-27B-FP8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="RedHatAI/Qwen3.6-27B-FP8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForImageTextToText

processor = AutoProcessor.from_pretrained("RedHatAI/Qwen3.6-27B-FP8")
model = AutoModelForImageTextToText.from_pretrained("RedHatAI/Qwen3.6-27B-FP8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use RedHatAI/Qwen3.6-27B-FP8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "RedHatAI/Qwen3.6-27B-FP8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/Qwen3.6-27B-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/RedHatAI/Qwen3.6-27B-FP8

SGLang

How to use RedHatAI/Qwen3.6-27B-FP8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "RedHatAI/Qwen3.6-27B-FP8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/Qwen3.6-27B-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "RedHatAI/Qwen3.6-27B-FP8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/Qwen3.6-27B-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use RedHatAI/Qwen3.6-27B-FP8 with Docker Model Runner:
```
docker model run hf.co/RedHatAI/Qwen3.6-27B-FP8
```

alexmarques commited on 26 days ago

Commit

5ab0ba8

verified ·

1 Parent(s): b870275

Add files using upload-large-folder tool

Browse files

Files changed (30) hide show

layers-1.safetensors +3 -0
layers-10.safetensors +3 -0
layers-11.safetensors +3 -0
layers-13.safetensors +3 -0
layers-16.safetensors +3 -0
layers-18.safetensors +3 -0
layers-21.safetensors +3 -0
layers-22.safetensors +3 -0
layers-23.safetensors +3 -0
layers-24.safetensors +3 -0
layers-28.safetensors +3 -0
layers-31.safetensors +3 -0
layers-34.safetensors +3 -0
layers-37.safetensors +3 -0
layers-38.safetensors +3 -0
layers-4.safetensors +3 -0
layers-41.safetensors +3 -0
layers-44.safetensors +3 -0
layers-47.safetensors +3 -0
layers-48.safetensors +3 -0
layers-51.safetensors +3 -0
layers-52.safetensors +3 -0
layers-53.safetensors +3 -0
layers-54.safetensors +3 -0
layers-58.safetensors +3 -0
layers-60.safetensors +3 -0
layers-61.safetensors +3 -0
layers-63.safetensors +3 -0
layers-8.safetensors +3 -0
outside.safetensors +3 -0

layers-1.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9a2efd1048386e560c8241c7b972e213b516f0cbeafc4de6f486bd7faddb8c39
+size 383865448

layers-10.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ebc4fd84671c2ac2d8d7b8da3707ed5d30738771a22eecc883d707521ed513a0
+size 383865472

layers-11.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9daf0f0b763489c0869061669293f93213d6060b1b0c88061b53bb97f9082cd6
+size 372313760

layers-13.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:61c0a820b09d58e45e516c56532007155b0fd214c11fa8906640c9217da95c84
+size 383865472

layers-16.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:84776e1d0a62c40730a4bc0f0e2c68242ba9a6150975d8992747fb6ecc0ebfcc
+size 383865472

layers-18.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1c6aed5f416dee34bb7ccfc322e0aa33f192b40afeab4b69fce784b2b349e253
+size 383865472

layers-21.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c8a7696ad5cd1be016cfc553351ee4b588f3d972084dd47028db8dba9b113870
+size 383865472

layers-22.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:08481a6a46057d0bcdcd2598d4f67d5f821725fc483800cd360c2d7047907e08
+size 383865472

layers-23.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:13b2ba6536c77330fa1c40e91c4c707d5086803299e0f57dd03142bb28faaaa4
+size 372313760

layers-24.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0f7855ee6b018e707d015f445f7e8b9542003104dbd0436e83e8c0dec73bdd17
+size 383865472

layers-28.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f10aecd4f9800052bbf1ad2cca939eb44dc205d42ff546096ce9212bd6eae411
+size 383865472

layers-31.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2a2e87f569b5da06a87b83df628cb27f491f3dbed55ca82054820cb382cffd2e
+size 372313760

layers-34.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6e08147ff5c8e0d5d7207d62ede78d6194f1453165cfab940743ff327bb0e6e5
+size 383865472

layers-37.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d38c28d8c86cc147e5edc1935dec46687a6b3a8bc0d712e829003dd9cd4fab7d
+size 383865472

layers-38.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1628a41fa5d3dbcf2ffde423530a9e94e7228e247e922e2269bd348071a76365
+size 383865472

layers-4.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9eb31d85023776894347a56101ee067f369d0d3c024f1331f5c7cba1f45b6df4
+size 383865448

layers-41.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:80877a16835f7311a112ad0bbaf1d19997142ea3e1df95ffe446fd52ad1f80d8
+size 383865472

layers-44.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:673543b94e59003265ae648bba11389c6735ee3264ec164108b6db99f1219587
+size 383865472

layers-47.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6a8e0ddb389485965cd30bd0a90d1be5e1acb65cb6311ce337199f268c3120f6
+size 372313760

layers-48.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:70bd74dcdf4ea8b09e1ae3463977a05aba73f2c02397491bb641a0a3ca6832a3
+size 383865472

layers-51.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:511f64772c1a5a15292eb1c48166b9d206088aed283185e10877fc43308097bd
+size 372313760

layers-52.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:92c35a4e850b4c0b87a1eab42c47996504e062a8a11699ada20fcea0b88f36c7
+size 383865472

layers-53.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ab48cad204502ddbf17f5ee2d5e52457baac965300cc15e8a25f35c843b4d526
+size 383865472

layers-54.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2804cee95e18e9e516b9490c36ea7f41553a3fcb4455cf44a361d6086ed214a3
+size 383865472

layers-58.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c9858a0c4bd8cb980829a45d3f9ca52a0bd506c6a0775eb179d3e5ebfaae7867
+size 383865472

layers-60.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1f24aa7a99e72f658089462b258613b11d06d6e11733ff5d5f864c663a190f92
+size 383865472

layers-61.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0b083f4afca3538f87545d4fc50817292fc80061a731a5f0235f5c406be87b45
+size 383865472

layers-63.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:7e27ec71df803eb38ba93aead54daaf1110d3f7124416443a30a3c81abc1c805
+size 372313760

layers-8.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:81ebe095241562b2e2ace8875e2771f0cdb68e76286a18506ea19a749b8de9b4
+size 383865448

outside.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:27a91100d904f6acc1c86d08675691199e1fe2da5da613106fb01ae4809b3ac1
+size 6007102112