BoKenlm-syl-v0.1 - Tibetan KenLM Language Model

A KenLM n-gram language model trained on Tibetan text, tokenized with syllable tokenizer.

Model Details

Parameter	Value
Model Type	Modified Kneser-Ney 5-gram
Tokenizer	Tibetan syllable-based (botok-rs SimpleTokenizer)
Training Corpus	bo_corpus.txt
Pruning	0 0 1
Tokens	153,703,256
Vocabulary Size	105,195

N-gram Statistics

Order	Count	D1	D2	D3+
1	105,195	0.7318	1.0392	1.3101
2	2,503,807	0.6749	1.0227	1.3576
3	6,956,090	0.7422	1.0862	1.3739
4	13,423,388	0.8156	1.1560	1.3755
5	16,307,695	0.7422	1.2835	1.5109

Memory Estimates

Type	MB	Details
probing	806	assuming -p 1.5
probing	937	assuming -r models -p 1.5
trie	379	without quantization
trie	206	assuming -q 8 -b 8 quantization
trie	334	assuming -a 22 array pointer compression
trie	161	assuming -a 22 -q 8 -b 8 array pointer compression and quantization

Training Resources

Metric	Value
Peak Virtual Memory	12,333 MB
Peak RSS	8,008 MB
Wall Time	88.5s
User Time	88.9s
System Time	26.3s

Usage

import kenlm

model = kenlm.Model("BoKenlm-syl-v0.1.arpa")

# Score a tokenized sentence
score = model.score("བོད་ སྐད་ ཀྱི་ ཚིག་ གྲུབ་ འདི་ ཡིན།")
print(score)

Files

• BoKenlm-syl-v0.1.arpa — ARPA format language model
• README.md — This model card

License

Apache 2.0