Fix transformers syntax.
Browse files
README.md
CHANGED
|
@@ -39,9 +39,9 @@ The tokenizer can be used directly via the `tokenizers` library or wrapped for u
|
|
| 39 |
|
| 40 |
### With `transformers`
|
| 41 |
|
| 42 |
-
```
|
| 43 |
from transformers import AutoTokenizer
|
| 44 |
|
| 45 |
tok = AutoTokenizer.from_pretrained("divilian/lilchatbot-tokenizer")
|
| 46 |
|
| 47 |
-
print(tok.
|
|
|
|
| 39 |
|
| 40 |
### With `transformers`
|
| 41 |
|
| 42 |
+
```
|
| 43 |
from transformers import AutoTokenizer
|
| 44 |
|
| 45 |
tok = AutoTokenizer.from_pretrained("divilian/lilchatbot-tokenizer")
|
| 46 |
|
| 47 |
+
print(tok.decode(tok("The lion's well-being matters — don’t forget that!").input_ids))
|