CoLA-Fisher-All-Attention-Q_K_V_O-seed30

This model is a fine-tuned version of roberta-base on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.4242
• Matthews Correlation: 0.5925

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 5

Training results

Training Loss	Epoch	Step	Validation Loss	Matthews Correlation
0.6313	0.1866	50	0.5814	0.0
0.5259	0.3731	100	0.4720	0.4531
0.478	0.5597	150	0.4844	0.4525
0.4601	0.7463	200	0.4399	0.5079
0.4253	0.9328	250	0.5527	0.4530
0.4308	1.1194	300	0.4239	0.5271
0.3956	1.3060	350	0.4998	0.4865
0.3959	1.4925	400	0.4725	0.4778
0.426	1.6791	450	0.4115	0.5573
0.3774	1.8657	500	0.5285	0.4996
0.3625	2.0522	550	0.5133	0.5430
0.3523	2.2388	600	0.5114	0.5426
0.3745	2.4254	650	0.4466	0.5335
0.3802	2.6119	700	0.4411	0.5554
0.3341	2.7985	750	0.4242	0.5925
0.3421	2.9851	800	0.4673	0.5495
0.3444	3.1716	850	0.4206	0.5751
0.3277	3.3582	900	0.4314	0.5708
0.3337	3.5448	950	0.5423	0.5339
0.3303	3.7313	1000	0.4399	0.5833
0.3219	3.9179	1050	0.4600	0.5858
0.322	4.1045	1100	0.4502	0.5881
0.2858	4.2910	1150	0.4870	0.5855
0.2938	4.4776	1200	0.4885	0.5856
0.3099	4.6642	1250	0.4640	0.5883
0.3057	4.8507	1300	0.4800	0.5779

Framework versions

• PEFT 0.16.0
• Transformers 4.54.1
• Pytorch 2.5.1+cu121
• Datasets 4.0.0
• Tokenizers 0.21.4