Instructions to use RedHatAI/DeepSeek-V4-Flash-BF16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use RedHatAI/DeepSeek-V4-Flash-BF16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="RedHatAI/DeepSeek-V4-Flash-BF16")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("RedHatAI/DeepSeek-V4-Flash-BF16")
model = AutoModelForCausalLM.from_pretrained("RedHatAI/DeepSeek-V4-Flash-BF16")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use RedHatAI/DeepSeek-V4-Flash-BF16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "RedHatAI/DeepSeek-V4-Flash-BF16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/DeepSeek-V4-Flash-BF16",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/RedHatAI/DeepSeek-V4-Flash-BF16

SGLang

How to use RedHatAI/DeepSeek-V4-Flash-BF16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "RedHatAI/DeepSeek-V4-Flash-BF16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/DeepSeek-V4-Flash-BF16",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "RedHatAI/DeepSeek-V4-Flash-BF16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/DeepSeek-V4-Flash-BF16",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use RedHatAI/DeepSeek-V4-Flash-BF16 with Docker Model Runner:
```
docker model run hf.co/RedHatAI/DeepSeek-V4-Flash-BF16
```

kylesayrs commited on 7 days ago

Commit

115582d

verified ·

1 Parent(s): fcd4ae8

Upload folder using huggingface_hub

Browse files

Files changed (14) hide show

model-00001-of-00013.safetensors +2 -2
model-00002-of-00013.safetensors +2 -2
model-00003-of-00013.safetensors +2 -2
model-00004-of-00013.safetensors +2 -2
model-00005-of-00013.safetensors +2 -2
model-00006-of-00013.safetensors +2 -2
model-00007-of-00013.safetensors +2 -2
model-00008-of-00013.safetensors +2 -2
model-00009-of-00013.safetensors +2 -2
model-00010-of-00013.safetensors +2 -2
model-00011-of-00013.safetensors +2 -2
model-00012-of-00013.safetensors +2 -2
model-00013-of-00013.safetensors +2 -2
model.safetensors.index.json +0 -0

model-00001-of-00013.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8ee3585fc6955d3f3f7f865030a0b01b3f28804d02e1e6acf07cbe3fa34bfafa
-size 45936952700

 version https://git-lfs.github.com/spec/v1
+oid sha256:8fa607c83718ed0698476e6f909cccf3881ccca2ba192f785500dfda57af7109
+size 45936936710

model-00002-of-00013.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c0776f7334d09bffc815ce393090a0be803179002f3b74a0d0d28ad5c08e294c
-size 48416904112

 version https://git-lfs.github.com/spec/v1
+oid sha256:a6cb015f5631aed0270ff4ed704a4db314263d8dace1b694c3142d6722848d28
+size 48416886204

model-00003-of-00013.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:54038e68aedb08ffdb5345e0b6c8b26bc0bc95f0e0b64f97276a7b52d21ebde2
-size 43814102880

 version https://git-lfs.github.com/spec/v1
+oid sha256:845bd4ba726829612fa97a7f5afecbbadfd800d9f59f67904f0d8a2a30df32e4
+size 43814086807

model-00004-of-00013.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:72f053ff91d654ed2594c9fcf3b2894ea5aac6e26119718fc6677f199db78d3a
-size 48416907032

 version https://git-lfs.github.com/spec/v1
+oid sha256:1bf1aabc916cfcefe26ddb728f93bac7998e1472d1a92b7bab937c77eb84114c
+size 48416889124

model-00005-of-00013.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6a9b99fe702a0002314640dd823438cbb4af3ad34893a42861d1a5d9389f883e
-size 43843749064

 version https://git-lfs.github.com/spec/v1
+oid sha256:60a001241988da9ae1ec2761f964a343b9d728de4a0023ccfed3224639a471d7
+size 43843732899

model-00006-of-00013.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c051347addd7f223476e4e539adbeba38791dc4747fed53813a78ab676e5ce8b
-size 48416907040

 version https://git-lfs.github.com/spec/v1
+oid sha256:0fe817630c3133e1034e312ccf76bf34de06bfc2c14b27adb21844674657883b
+size 48416889132

model-00007-of-00013.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:38c820e3e1d9dafa1ce1226054c0f633531e92618f361f6b5996a4edc4e2f0bf
-size 43814105256

 version https://git-lfs.github.com/spec/v1
+oid sha256:c6f6d55a5ef0a495809bc0348d38c67321c23ec9014dac1f66d64182912e703d
+size 43814089179

model-00008-of-00013.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4b171d9f85d46b634dd00683a3e1683fdc12acd4398f09a4be7fa65bd289a548
-size 48416907032

 version https://git-lfs.github.com/spec/v1
+oid sha256:558d1ac8b46d20a91f371b71e0df585c8555d144a8001300e849f6319a3b67ed
+size 48416889124

model-00009-of-00013.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b2962dabdc2469f09674542d512bbf7554669db0493f7e5cad4b66008419840f
-size 43843749064

 version https://git-lfs.github.com/spec/v1
+oid sha256:6c7b360517c7d3bbfb4fc70dfdf41c8e624acba562ff5ca844dafd35ed9ff78d
+size 43843732899

model-00010-of-00013.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ae6a55e88a011970c8b3a3fd133823748fef09914014fc98612bec59fef6d35f
-size 48416907040

 version https://git-lfs.github.com/spec/v1
+oid sha256:415614dd7281d101853ecbccf27e59cfe69b52162fe94491e9385be51b489285
+size 48416889132

model-00011-of-00013.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:79665068dafdf611b7b0f4f4ac0fcbf044c4293ddae931877193f73c8893df6c
-size 43814105256

 version https://git-lfs.github.com/spec/v1
+oid sha256:6ac416a07598fcacb59dae3210ef87272ef8bde45ceaeb29d1b5e5b5bd4da61b
+size 43814089179

model-00012-of-00013.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:03a78e7e48c2de92cd844cb52c633e11dbd03741f581c5e56f7945fdaa581ac9
-size 48416907032

 version https://git-lfs.github.com/spec/v1
+oid sha256:f940f73cdfa30a2e6502072b6641a624d4032a566c304e1a98862309d102ad85
+size 48416889124

model-00013-of-00013.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:180c9e000684fa00a7e9ba77b47f2176db57f12a6174b9a647a7cfa998747a84
-size 13189650784

 version https://git-lfs.github.com/spec/v1
+oid sha256:0c87df6b43b2376e3afd322343496422296dee4ad7f1ad7bae6e2d5cda0a7395
+size 13189645885

model.safetensors.index.json CHANGED Viewed

The diff for this file is too large to render. See raw diff