nobinomial-babylm-base_seed-42_1e-3
This model was trained from scratch on an unknown dataset. It achieves the following results on the evaluation set:
- Loss: 3.1257
- Accuracy: 0.4109
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.001
- train_batch_size: 96
- eval_batch_size: 192
- seed: 42
- gradient_accumulation_steps: 8
- total_train_batch_size: 768
- optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 20.0
Training results
| Training Loss | Epoch | Step | Validation Loss | Accuracy |
|---|---|---|---|---|
| 3.903 | 1.0 | 553 | 3.7592 | 0.3401 |
| 3.3665 | 2.0 | 1106 | 3.3901 | 0.3762 |
| 3.2092 | 3.0 | 1659 | 3.2582 | 0.3899 |
| 3.0995 | 4.0 | 2212 | 3.1902 | 0.3975 |
| 3.0327 | 5.0 | 2765 | 3.1513 | 0.4019 |
| 2.9704 | 6.0 | 3318 | 3.1253 | 0.4047 |
| 2.9274 | 7.0 | 3871 | 3.1099 | 0.4068 |
| 2.8872 | 8.0 | 4424 | 3.1010 | 0.4082 |
| 2.846 | 9.0 | 4977 | 3.0952 | 0.4093 |
| 2.8121 | 10.0 | 5530 | 3.0940 | 0.4101 |
| 2.7761 | 11.0 | 6083 | 3.0902 | 0.4109 |
| 2.7405 | 12.0 | 6636 | 3.0967 | 0.4110 |
| 2.7054 | 13.0 | 7189 | 3.1030 | 0.4112 |
| 2.673 | 14.0 | 7742 | 3.1043 | 0.4112 |
| 2.6377 | 15.0 | 8295 | 3.1180 | 0.4106 |
| 2.6077 | 16.0 | 8848 | 3.1257 | 0.4109 |
Framework versions
- Transformers 4.57.3
- Pytorch 2.6.0+cu124
- Datasets 4.4.1
- Tokenizers 0.22.1
- Downloads last month
- 4