Grokking at the Edge of Numerical Stability
Paper • 2501.04697 • Published • 2
Fine-tuned/hyperfitted with methodology from https://arxiv.org/abs/2412.04318
using OrthoGrad optimizer https://arxiv.org/abs/2501.04697
Updated 23.02.2025: same dataset, 512 token sequences with 64 tokens sliding window (loss still decreased). Significant hellaswag drop (~22%)
- Downloads last month
- 4
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support