# LilChatBot WordLevel Tokenizer

A **WordLevel tokenizer** trained for the *LilChatBot* project.

This tokenizer is designed for clarity, interpretability, and stability rather than maximum compression. It is intended primarily for educational and experimental language-model work.

---

## Design choices

- **WordLevel tokenization** (no subword splitting)
- **Lowercasing**
- **Unicode NFKC normalization**
- **Apostrophes preserved everywhere**  
  (e.g. `don't`, `lion's`, `'hello'`, `James'`)
- **Aggressive punctuation isolation**, including:
  - sentence punctuation (`. , ! ? ; :`)
  - brackets (`() [] {}`)
  - slashes (`/`)
  - double quotes (straight and curly)
  - en/em dashes (`– —`)
- **Repeated punctuation collapsed**  
  (`!!! → !`, `??? → ?`, `... → .`)
- English-focused

This tokenizer intentionally favors **lexical transparency** over vocabulary compactness.

---

## Files

- `tokenizer.json` — complete tokenizer definition (normalizer, pre-tokenizer, vocab, special tokens)

The tokenizer can be used directly via the `tokenizers` library or wrapped for use with `transformers`.

---

## Usage

### With `transformers`

```
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("divilian/lilchatbot-tokenizer")

print(tok.decode(tok("The lion's well-being matters — don’t forget that!").input_ids))