Instructions to use OsaurusAI/DeepSeek-V4-Flash-JANGTQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use OsaurusAI/DeepSeek-V4-Flash-JANGTQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="OsaurusAI/DeepSeek-V4-Flash-JANGTQ")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("OsaurusAI/DeepSeek-V4-Flash-JANGTQ")
model = AutoModelForCausalLM.from_pretrained("OsaurusAI/DeepSeek-V4-Flash-JANGTQ")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use OsaurusAI/DeepSeek-V4-Flash-JANGTQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "OsaurusAI/DeepSeek-V4-Flash-JANGTQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OsaurusAI/DeepSeek-V4-Flash-JANGTQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/OsaurusAI/DeepSeek-V4-Flash-JANGTQ

SGLang

How to use OsaurusAI/DeepSeek-V4-Flash-JANGTQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "OsaurusAI/DeepSeek-V4-Flash-JANGTQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OsaurusAI/DeepSeek-V4-Flash-JANGTQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "OsaurusAI/DeepSeek-V4-Flash-JANGTQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OsaurusAI/DeepSeek-V4-Flash-JANGTQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use OsaurusAI/DeepSeek-V4-Flash-JANGTQ with Docker Model Runner:
```
docker model run hf.co/OsaurusAI/DeepSeek-V4-Flash-JANGTQ
```

Osaurus-AI commited on 1 day ago

Commit

cec83e9

verified ·

1 Parent(s): 9a4bae5

Upload jang_config.json with huggingface_hub

Browse files

Files changed (1) hide show

jang_config.json +73 -0

jang_config.json ADDED Viewed

	@@ -0,0 +1,73 @@

+{
+  "weight_format": "mxtq",
+  "profile": "JANGTQ_V3",
+  "variant": "V3",
+  "mxtq_seed": 42,
+  "drop_mtp": true,
+  "source_model": "/Users/eric/models/Sources/DeepSeek-V4-Flash",
+  "source_config": {
+    "n_routed_experts": 256,
+    "num_hidden_layers": 43,
+    "n_hash_layers": 3
+  },
+  "mxtq_bits": {
+    "routed_expert": 2,
+    "attention": 8,
+    "shared_expert": 8,
+    "compressor": 8,
+    "indexer": 8,
+    "embed_tokens": 8,
+    "lm_head": 8,
+    "norms_router_hc": 16
+  },
+  "model_family": "deepseek_v4",
+  "chat": {
+    "encoder": "encoding_dsv4",
+    "encoder_fn": "encode_messages",
+    "chat_template_source": "builtin_encoding_module",
+    "has_tokenizer_chat_template": false,
+    "bos_token": "<\uff5cbegin\u2581of\u2581sentence\uff5c>",
+    "eos_token": "<\uff5cend\u2581of\u2581sentence\uff5c>",
+    "bos_token_id": 0,
+    "eos_token_id": 1,
+    "role_tokens": {
+      "user": "<\uff5cUser\uff5c>",
+      "assistant": "<\uff5cAssistant\uff5c>",
+      "latest_reminder": "<\uff5clatest_reminder\uff5c>"
+    },
+    "reasoning": {
+      "supported": true,
+      "modes": [
+        "chat",
+        "thinking"
+      ],
+      "default_mode": "chat",
+      "thinking_start": "<think>",
+      "thinking_end": "</think>",
+      "reasoning_effort_levels": [
+        "max",
+        "high",
+        null
+      ],
+      "drop_earlier_reasoning": true
+    },
+    "tool_calling": {
+      "supported": true,
+      "parser": "dsml",
+      "dsml_token": "\uff5cDSML\uff5c",
+      "tool_calls_block": "tool_calls",
+      "invoke_block": "invoke",
+      "parameter_block": "parameter",
+      "tool_output_tag": "tool_result"
+    },
+    "sampling_defaults": {
+      "temperature": 0.6,
+      "top_p": 0.95,
+      "repetition_penalty": 1.0,
+      "repetition_penalty_thinking": 1.0,
+      "repetition_penalty_chat": 1.05,
+      "max_new_tokens": 4096
+    }
+  },
+  "routed_expert_layout": "prestacked"
+}