pipeline_detokenization on Windows
#13
by Anthonyy232 - opened
Pretty sure pipeline_detokenization tweaks out and doesn't work on Windows because no fork()
Hi! We recommend you try the huggingface native implementation in transformers >=5.4.0. The tokenizer is much faster now which means we don't need pipeline_detokenization (and this option isn't present in the transformers version)