divilian commited on
Commit
9699209
·
1 Parent(s): 03fa779

Fix transformers syntax.

Browse files
Files changed (1) hide show
  1. README.md +2 -2
README.md CHANGED
@@ -39,9 +39,9 @@ The tokenizer can be used directly via the `tokenizers` library or wrapped for u
39
 
40
  ### With `transformers`
41
 
42
- ```python
43
  from transformers import AutoTokenizer
44
 
45
  tok = AutoTokenizer.from_pretrained("divilian/lilchatbot-tokenizer")
46
 
47
- print(tok.encode("The lions well-being matters — don't forget that!").tokens)
 
39
 
40
  ### With `transformers`
41
 
42
+ ```
43
  from transformers import AutoTokenizer
44
 
45
  tok = AutoTokenizer.from_pretrained("divilian/lilchatbot-tokenizer")
46
 
47
+ print(tok.decode(tok("The lion's well-being matters — dont forget that!").input_ids))