Hilberg Scaling in Neural Language Models (ICSDS 2025)
Collection
A suite of three GPT autoregressive language models trained on the July 20, 2025 English Wikipedia dump for experiments on entropy scaling. • 4 items • Updated
GPT-Hilberg-5M
The present is a 5M-parameter GPT autoregressive language model trained on the July 20, 2025 English Wikipedia dump for experiments on entropy scaling and Hilberg conjecture. For more information on this, you can check here. Dataset available here.
This model is part of the following suite:
- GPT-Hilberg-1M (128 ctx)
- GPT-Hilberg-5M (512 ctx)
- GPT-Hilberg-10M (1024 ctx)
Architecture details
- Parameters: ~5M
- Layers: 10
- Model dimension: 128
- Attention heads: 8
- FF dimension: 512
- Context length: 512
- Vocabulary size: 16,000
Training details
- Dataset: English Wikipedia (2025-07 dump)
- Tokenizer: SentencePiece BPE
- Purpose: Autogregression
- Optimizer: AdamW
- Downloads last month
- 111
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support