🧠✨ TransformerLM (AR 512, 8K vocab) — SimpleStories
This is the result of the code from https://github.com/triloy8/transformerlm, a minimal autoregressive Transformer LM trained on SimpleStories with a 512-token context and an 8K vocab tokenizer. ✨
✅ Key Facts
- Model type: Autoregressive Transformer LM
- Dataset: SimpleStories
- Context length: 512 tokens
- Tokenizer vocab size: 8,000
- Layers: 12
- Heads: 8
- d_model: 512
- d_ff: 2,048
- Training setup: Single NVIDIA A40 48GB
- Runtime: ~20 hours ⏱️
📦 What’s Inside
- 60k steps from a 60k run, including:
- Optimizer state
- RNG state
- Safetensors weights
- Tokenizer config
- Run config
🚀 Reproducibility
To reproduce the run:
Exact commit that launched the train: https://github.com/triloy8/transformerlm/commit/06cb4831d47c04a18573bee8e28dc83b10086d06