privacy-filter-nemotron / tokenizer_config.json

Initial upload: privacy-filter fine-tuned for 55 Nemotron-PII labels

61f8bf4 verified 13 days ago

234 Bytes

	{
	"backend": "tokenizers",
	"eos_token": "<\|endoftext\|>",
	"model_input_names": [
	"input_ids",
	"attention_mask"
	],
	"model_max_length": 128000,
	"pad_token": "<\|endoftext\|>",
	"tokenizer_class": "TokenizersBackend"
	}