corex_tok_info.txt

CoreX Tokenizer Information
==========================
Vocabulary Size: 32000
Model Type: unigram
Special Tokens:
  PAD: 0 -> '<pad>'
  UNK: 1 -> '<unk>'
  BOS: 2 -> '<s>'
  EOS: 3 -> '</s>'