onnx-magpie / tokenizers /french_chartokenizer.json
mweinbach1's picture
Add files using upload-large-folder tool
2b9fdf9 verified
raw
history blame contribute delete
219 Bytes
{
"kind": "byt5",
"class_name": "ByT5Tokenizer",
"offset": 633,
"pad_id": 633,
"num_tokens": 384,
"aggregate_offset": 633,
"byte_offset": 3,
"eos_token_id": 1,
"global_eos_token_id": 2361
}