opt-babylm2-rewritten-clean-spacy-earlystop_hierarchical_211_size-origin_both-bpe_seed-1024_1e-3

This model was trained from scratch on the kanishka/babylm2-rewritten-clean-spacy_hierarchical-adj_211_size-origin_both-ablation dataset. It achieves the following results on the evaluation set:

• Loss: 2.6833
• Accuracy: 0.4786

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.001
• train_batch_size: 32
• eval_batch_size: 64
• seed: 1024
• gradient_accumulation_steps: 8
• total_train_batch_size: 256
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 32000
• num_epochs: 20.0
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
4.0984	0.9997	2255	3.8123	0.3607
3.451	1.9997	4510	3.2992	0.4095
3.1285	2.9997	6765	3.0860	0.4308
2.9204	3.9997	9020	2.9801	0.4413
2.8436	4.9997	11275	2.9212	0.4476
2.7858	5.9997	13530	2.8823	0.4514
2.7439	6.9997	15785	2.8541	0.4549
2.7113	7.9997	18040	2.8362	0.4568
2.6881	8.9997	20295	2.8207	0.4588
2.6666	9.9997	22550	2.8107	0.4597
2.6497	10.9997	24805	2.8031	0.4610
2.6483	11.9997	27060	2.7951	0.4617
2.6384	12.9997	29315	2.7930	0.4619
2.6268	13.9997	31570	2.7862	0.4628
2.6122	14.9997	33825	2.7651	0.4654
2.5657	15.9997	36080	2.7415	0.4684
2.5189	16.9997	38335	2.7190	0.4718
2.4616	17.9997	40590	2.6998	0.4746
2.3977	18.9997	42845	2.6860	0.4772
2.3247	19.9997	45100	2.6833	0.4786

Framework versions

• Transformers 4.48.0
• Pytorch 2.6.0+cu124
• Datasets 3.2.0
• Tokenizers 0.21.4