# Test-shakespeare

Trained with **transformer-toolkit**.

## Architecture
| param | value |
|---|---|
| `vocab_size` | `32000` |
| `dim` | `512` |
| `n_layers` | `12` |
| `n_heads` | `8` |
| `max_seq` | `512` |
| `attn` | `gqa` |
| `n_kv_heads` | `2` |
| `latent_dim` | `64` |
| `ffn` | `swiglu` |
| `hidden_dim` | `1536` |
| `n_experts` | `8` |
| `top_k` | `2` |
| `moe_aux_weight` | `0.01` |
| `moe_capacity` | `1.0` |
| `moe_n_shared` | `2` |
| `moe_n_routed` | `6` |
| `norm` | `rmsnorm` |
| `eps` | `1e-06` |
| `pos_enc` | `rope` |
| `dropout` | `0.1` |
| `tie_weights` | `False` |

## Metrics
| metric | value |
|---|---|
| `val_loss` | `3.949221694469452` |