Instructions to use LLM-OS-Models/HRM-Text-Ko-Terminal-Tokenizer-131K with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use LLM-OS-Models/HRM-Text-Ko-Terminal-Tokenizer-131K with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="LLM-OS-Models/HRM-Text-Ko-Terminal-Tokenizer-131K")

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("LLM-OS-Models/HRM-Text-Ko-Terminal-Tokenizer-131K", dtype="auto")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use LLM-OS-Models/HRM-Text-Ko-Terminal-Tokenizer-131K with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "LLM-OS-Models/HRM-Text-Ko-Terminal-Tokenizer-131K"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LLM-OS-Models/HRM-Text-Ko-Terminal-Tokenizer-131K",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/LLM-OS-Models/HRM-Text-Ko-Terminal-Tokenizer-131K

SGLang

How to use LLM-OS-Models/HRM-Text-Ko-Terminal-Tokenizer-131K with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "LLM-OS-Models/HRM-Text-Ko-Terminal-Tokenizer-131K" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LLM-OS-Models/HRM-Text-Ko-Terminal-Tokenizer-131K",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "LLM-OS-Models/HRM-Text-Ko-Terminal-Tokenizer-131K" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LLM-OS-Models/HRM-Text-Ko-Terminal-Tokenizer-131K",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use LLM-OS-Models/HRM-Text-Ko-Terminal-Tokenizer-131K with Docker Model Runner:
```
docker model run hf.co/LLM-OS-Models/HRM-Text-Ko-Terminal-Tokenizer-131K
```

gyung commited on 1 day ago

Commit

857a0d8

verified ·

1 Parent(s): f9f5786

Upload HRM Ko-Terminal tokenizer v1

Browse files

Files changed (4) hide show

.gitattributes +1 -0
README.md +42 -0
tokenizer.json +3 -0
tokenizer_training_manifest.json +66 -0

.gitattributes CHANGED Viewed

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text

 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md ADDED Viewed

	@@ -0,0 +1,42 @@

+# HRM Ko-Terminal 131K Tokenizer v1
+Built on 2026-05-23 for HRM-Text Korean terminal/tool-call pre-training.
+## Training
+- Algorithm: byte-level BPE
+- Vocabulary size: 131,072
+- Normalization: NFC
+- Corpus cap: 2.5GiB total, 256MiB per top-level input source
+- Goal: Korean, English, code, terminal commands, JSON/tool-call formats
+## Efficiency Check
+| sample | chars/token |
+|---|---:|
+| Korean general | 2.60 |
+| Korean legal | 2.36 |
+| Korean terminal instruction | 2.18 |
+| Shell command | 2.68 |
+| Tool-call JSON | 3.32 |
+| Python code | 3.37 |
+| English | 4.40 |
+Core HRM/chat/tool special tokens encode as single tokens:
+- `<|im_start|>`
+- `<|im_end|>`
+- `<|assistant|>`
+- `<|tool_call|>`
+- `<|terminal|>`
+- `<|box_end|>`
+## Notes
+This tokenizer keeps HRM-Text control tokens used by `scripts/prepare_sft_data.py`,
+including the default condition mapping:
+- `direct=<|object_ref_start|>`
+- `cot=<|object_ref_end|>`
+- `noisy=<|quad_start|>`
+- `synth=<|quad_end|>`

tokenizer.json ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ef6d5204ebfb25e992926714af88ad6b77e12a90ea6f3eb0f200e1a1f8712d5c
+size 11457812

tokenizer_training_manifest.json ADDED Viewed

	@@ -0,0 +1,66 @@

+{
+  "vocab_size": 131072,
+  "requested_vocab_size": 131072,
+  "max_gib": 2.5,
+  "max_mib_per_input": 256.0,
+  "min_frequency": 2,
+  "special_tokens": [
+    "<|PAD|>",
+    "<|unk|>",
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|system|>",
+    "<|user|>",
+    "<|assistant|>",
+    "<|tool_call|>",
+    "<|/tool_call|>",
+    "<|tool_response|>",
+    "<|function|>",
+    "<|/function|>",
+    "<|execute|>",
+    "<|result|>",
+    "<|terminal|>",
+    "<|/terminal|>",
+    "<|command|>",
+    "<|output|>",
+    "<|error|>",
+    "<|exit_code|>",
+    "<|json_start|>",
+    "<|json_end|>",
+    "<|xml_start|>",
+    "<|xml_end|>",
+    "<|code_start|>",
+    "<|code_end|>",
+    "<think>",
+    "</think>",
+    "<|direct|>",
+    "<|cot|>",
+    "<|noisy|>",
+    "<|synth|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>",
+    "<|fim_prefix|>",
+    "<|fim_middle|>",
+    "<|fim_suffix|>"
+  ],
+  "inputs": [
+    "HRM-Text/legalize-kr",
+    "HRM-Text/ordinance-kr",
+    "admrule-kr",
+    "precedent-kr",
+    "dataset",
+    "HRM-Text/data_toolbench/data",
+    "/home/work/.data/huggingface/hrm_text_extra/sft",
+    "/home/work/.data/huggingface/hrm_text_extra/tokenizer_corpus",
+    "/home/work/.data/huggingface/hrm_text_extra/raw/angrygiraffe__claude-opus-4.6-4.7-reasoning-8.7k"
+  ]
+}