tapt_ulmfit_reinit_whole_word_2K_no_reinit_classifier-LR_2e-05

This model is a fine-tuned version of microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.8732
• Accuracy: 0.6229

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 3407
• gradient_accumulation_steps: 2
• total_train_batch_size: 32
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-06 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.06
• num_epochs: 100
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
4.8478	1.0	21	4.1692	0.3728
4.3976	2.0	42	3.6013	0.4217
3.6892	3.0	63	3.0326	0.4720
3.218	4.0	84	2.7638	0.5086
2.9458	5.0	105	2.5383	0.5313
2.7757	6.0	126	2.5290	0.5363
2.6517	7.0	147	2.4508	0.5411
2.5866	8.0	168	2.4045	0.5495
2.5309	9.0	189	2.3862	0.5527
2.487	10.0	210	2.2580	0.5694
2.3918	11.0	231	2.2978	0.5657
2.3519	12.0	252	2.2710	0.5630
2.2949	13.0	273	2.2777	0.5676
2.2473	14.0	294	2.1882	0.5768
2.2523	15.0	315	2.1572	0.5810
2.2279	16.0	336	2.1637	0.5790
2.2047	17.0	357	2.1343	0.5845
2.1579	18.0	378	2.1103	0.5894
2.1077	19.0	399	2.1620	0.5817
2.1134	20.0	420	2.1590	0.5832
2.075	21.0	441	2.1596	0.5869
2.1034	22.0	462	2.1361	0.5837
2.0437	23.0	483	2.0547	0.5979
2.0201	24.0	504	2.0388	0.6019
2.0296	25.0	525	2.0262	0.5982
2.0156	26.0	546	2.0801	0.5979
2.0084	27.0	567	2.0284	0.6101
2.0238	28.0	588	2.0932	0.5956
1.9875	29.0	609	2.0586	0.6010
1.991	30.0	630	2.0285	0.6071
1.9585	31.0	651	2.0434	0.6057
1.9708	32.0	672	2.0445	0.6012
1.9386	33.0	693	2.0200	0.6021
1.9513	34.0	714	2.0381	0.6046
1.9331	35.0	735	2.0244	0.6048
1.9136	36.0	756	2.0106	0.6103
1.8936	37.0	777	1.9835	0.6108
1.9083	38.0	798	2.0139	0.6091
1.9366	39.0	819	1.9732	0.6094
1.9054	40.0	840	2.0056	0.6117
1.8806	41.0	861	1.9351	0.6182
1.8945	42.0	882	1.9703	0.6117
1.8826	43.0	903	1.9916	0.6101
1.9384	44.0	924	1.9139	0.6163
1.8644	45.0	945	1.9324	0.6176
1.835	46.0	966	1.9390	0.6155
1.8745	47.0	987	1.9310	0.6245
1.8827	48.0	1008	1.9099	0.6211
1.8579	49.0	1029	1.9579	0.6143
1.8517	50.0	1050	1.9367	0.6175
1.8561	51.0	1071	1.9641	0.6166
1.8475	52.0	1092	1.9482	0.6159
1.8312	53.0	1113	1.9906	0.6153
1.8315	54.0	1134	1.9376	0.6202
1.8357	55.0	1155	1.9470	0.6183
1.8385	56.0	1176	1.9325	0.6164
1.836	57.0	1197	1.9579	0.6150
1.881	58.0	1218	1.9786	0.6083
1.8277	59.0	1239	1.8786	0.6274
1.8371	60.0	1260	1.9729	0.6107
1.8089	61.0	1281	1.9560	0.6154
1.8252	62.0	1302	1.9502	0.6239
1.8189	63.0	1323	1.9162	0.6212
1.8182	64.0	1344	1.9562	0.6137
1.8408	65.0	1365	1.9497	0.6166
1.7901	66.0	1386	1.9537	0.6171
1.8303	67.0	1407	1.9439	0.6190
1.7722	68.0	1428	1.9331	0.6156
1.7553	69.0	1449	1.9355	0.6161
1.7821	70.0	1470	1.8997	0.6176
1.7935	71.0	1491	1.9795	0.6153
1.801	72.0	1512	1.9667	0.6199
1.8056	73.0	1533	1.8894	0.625
1.788	74.0	1554	1.9302	0.6185
1.777	75.0	1575	1.8276	0.6301
1.7724	76.0	1596	1.9922	0.6126
1.7637	77.0	1617	1.8786	0.6286
1.7892	78.0	1638	1.8821	0.6272
1.7662	79.0	1659	1.9107	0.6227
1.7938	80.0	1680	1.9456	0.6227
1.7794	81.0	1701	1.8990	0.6244
1.7758	82.0	1722	1.9226	0.6179
1.756	83.0	1743	1.9059	0.6218
1.7955	84.0	1764	1.9066	0.6200
1.7639	85.0	1785	1.9427	0.6123
1.759	86.0	1806	1.8793	0.6251
1.779	87.0	1827	1.8987	0.6285
1.7953	88.0	1848	1.8638	0.6255
1.7517	89.0	1869	1.9097	0.6215
1.7635	90.0	1890	1.8898	0.6232
1.7694	91.0	1911	1.8791	0.6244
1.7466	92.0	1932	1.8727	0.6245
1.7581	93.0	1953	1.8948	0.6261
1.7752	94.0	1974	1.9361	0.6177
1.7866	95.0	1995	1.9210	0.6201
1.7677	96.0	2016	1.8401	0.6268
1.7656	97.0	2037	1.9306	0.6243
1.749	98.0	2058	1.8680	0.6269
1.7457	99.0	2079	1.8673	0.6302
1.7603	100.0	2100	1.8732	0.6229

Framework versions

• Transformers 4.48.2
• Pytorch 2.4.1+cu121
• Datasets 3.0.2
• Tokenizers 0.21.0