opt-babylm2-rewritten-clean-spacy-earlystop_hierarchical_211_size-origin_adj1-bpe_seed-1024_1e-3

This model was trained from scratch on the kanishka/babylm2-rewritten-clean-spacy_hierarchical-adj_211_size-origin_adj1-ablation dataset. It achieves the following results on the evaluation set:

• Loss: 2.6825
• Accuracy: 0.4787

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.001
• train_batch_size: 32
• eval_batch_size: 64
• seed: 1024
• gradient_accumulation_steps: 8
• total_train_batch_size: 256
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 32000
• num_epochs: 20.0
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
4.0976	0.9996	2255	3.8134	0.3609
3.451	1.9996	4510	3.2975	0.4099
3.1296	2.9996	6765	3.0861	0.4309
2.9216	3.9996	9020	2.9813	0.4414
2.8416	4.9996	11275	2.9210	0.4476
2.7862	5.9996	13530	2.8825	0.4515
2.7433	6.9996	15785	2.8527	0.4548
2.7114	7.9996	18040	2.8345	0.4570
2.6882	8.9996	20295	2.8197	0.4588
2.6689	9.9996	22550	2.8123	0.4599
2.6501	10.9996	24805	2.8021	0.4608
2.6475	11.9996	27060	2.7964	0.4617
2.6373	12.9996	29315	2.7920	0.4621
2.6283	13.9996	31570	2.7889	0.4624
2.6117	14.9996	33825	2.7657	0.4654
2.568	15.9996	36080	2.7407	0.4686
2.519	16.9996	38335	2.7193	0.4717
2.463	17.9996	40590	2.7002	0.4747
2.3979	18.9996	42845	2.6854	0.4774
2.3268	19.9996	45100	2.6825	0.4787

Framework versions

• Transformers 4.48.0
• Pytorch 2.6.0+cu124
• Datasets 3.2.0
• Tokenizers 0.21.4