onnx-magpie / tokenizers /text_ce_tokenizer.json
mweinbach1's picture
Add files using upload-large-folder tool
2b9fdf9 verified
raw
history blame contribute delete
222 Bytes
{
"kind": "byt5",
"class_name": "ByT5Tokenizer",
"offset": 1976,
"pad_id": 1976,
"num_tokens": 384,
"aggregate_offset": 1976,
"byte_offset": 3,
"eos_token_id": 1,
"global_eos_token_id": 2361
}