llama3.2-1B-pt/README.md

metadata

library_name: transformers
license: other
base_model: meta-llama/Llama-3.2-1B
tags:
  - llama-factory
  - full
  - generated_from_trainer
model-index:
  - name: llama3.2-1B-pt
    results: []

llama3.2-1B-pt

This model is a fine-tuned version of meta-llama/Llama-3.2-1B on the dolma_v17 dataset. It achieves the following results on the evaluation set:

• Loss: 3.3814

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• total_train_batch_size: 16
• total_eval_batch_size: 16
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 2000
• training_steps: 480000

Training results

Training Loss	Epoch	Step	Validation Loss
4.141	0.0021	1000	4.0261
4.5266	0.0042	2000	4.5481
4.59	0.0063	3000	4.5945
4.4559	0.0083	4000	4.4359
4.4322	0.0104	5000	4.3558
4.2777	0.0125	6000	4.2885
4.2747	0.0146	7000	4.2384
4.2632	0.0167	8000	4.2193
4.2457	0.0187	9000	4.1833
4.1475	0.0208	10000	4.1386
4.141	0.0229	11000	4.0980
4.1236	0.025	12000	4.0966
4.1118	0.0271	13000	4.0820
4.1394	0.0292	14000	4.0277
4.006	0.0312	15000	4.0100
4.0928	0.0333	16000	4.0256
4.0917	0.0354	17000	4.0059
4.0418	0.0375	18000	3.9906
4.069	0.0396	19000	3.9773
3.9632	0.0417	20000	3.9625
4.1044	0.0437	21000	3.9744
4.0262	0.0458	22000	3.9423
4.0516	0.0479	23000	3.9458
4.0737	0.05	24000	3.9307
3.9643	0.0521	25000	3.9314
3.9472	0.0542	26000	3.9337
3.9465	0.0563	27000	3.9346
3.902	0.0583	28000	3.9161
4.0343	0.0604	29000	3.8905
3.9185	0.0625	30000	3.8983
3.8769	0.0646	31000	3.8749
3.9741	0.0667	32000	3.8434
3.8712	0.0688	33000	3.8822
3.7466	0.0708	34000	3.8275
3.9764	0.0729	35000	3.8520
3.8212	0.075	36000	3.8515
3.8671	0.0771	37000	3.8439
3.8405	0.0792	38000	3.8555
3.8123	0.0813	39000	3.8229
3.9641	0.0833	40000	3.8233
3.7526	0.0854	41000	3.8344
3.9086	0.0875	42000	3.8212
3.8417	0.0896	43000	3.8113
3.8525	0.0917	44000	3.7934
3.7642	0.0938	45000	3.8093
3.8702	0.0958	46000	3.7896
3.8191	0.0979	47000	3.7950
3.8887	0.1	48000	3.7916
3.8501	0.1021	49000	3.7954
3.7836	0.1042	50000	3.7698
3.8468	0.1062	51000	3.7577
3.8272	0.1083	52000	3.7868
3.8116	0.1104	53000	3.7769
3.7229	0.1125	54000	3.7917
3.7081	0.1146	55000	3.7711
3.8381	0.1167	56000	3.7697
3.7386	0.1187	57000	3.7559
3.75	0.1208	58000	3.7578
3.7611	0.1229	59000	3.7471
3.6357	0.125	60000	3.7345
3.8717	0.1271	61000	3.7425
3.7516	0.1292	62000	3.7501
3.6488	0.1313	63000	3.7434
3.7167	0.1333	64000	3.7177
3.6957	0.1354	65000	3.7139
3.693	0.1375	66000	3.7310
3.8045	0.1396	67000	3.7135
3.8231	0.1417	68000	3.7090
3.7704	0.1437	69000	3.7081
3.7848	0.1458	70000	3.7024
3.6931	0.1479	71000	3.6950
3.8635	0.15	72000	3.6928
3.8062	0.1521	73000	3.6915
3.6754	0.1542	74000	3.7145
3.8106	0.1562	75000	3.6927
3.7576	0.1583	76000	3.7038
3.7289	0.1604	77000	3.6871
3.7628	0.1625	78000	3.6730
3.6676	0.1646	79000	3.6735
3.5012	0.1667	80000	3.6606
3.4192	0.1688	81000	3.6842
3.4183	0.1708	82000	3.6978
3.4288	0.1729	83000	3.6978
3.3554	0.175	84000	3.6865
3.375	0.1771	85000	3.6816
3.3618	0.1792	86000	3.6732
3.4111	0.1812	87000	3.6747
3.4076	0.1833	88000	3.6576
3.321	0.1854	89000	3.6939
3.3093	0.1875	90000	3.6796
3.3298	0.1896	91000	3.6907
3.2924	0.1917	92000	3.6911
3.383	0.1938	93000	3.6845
3.3564	0.1958	94000	3.6801
3.3192	0.1979	95000	3.6735
3.3107	0.2	96000	3.6852
3.3519	0.2021	97000	3.6796
3.3198	0.2042	98000	3.6680
3.2356	0.2062	99000	3.6669
3.3781	0.2083	100000	3.6826
3.3048	0.2104	101000	3.6540
3.2758	0.2125	102000	3.6481
3.3009	0.2146	103000	3.6656
3.3017	0.2167	104000	3.6755
3.2664	0.2188	105000	3.6714
3.3034	0.2208	106000	3.6694
3.2943	0.2229	107000	3.6815
3.293	0.225	108000	3.6753
3.3475	0.2271	109000	3.6721
3.3305	0.2292	110000	3.6722
3.3044	0.2313	111000	3.6733
3.3002	0.2333	112000	3.6798
3.2265	0.2354	113000	3.6611
3.2664	0.2375	114000	3.6662
3.2878	0.2396	115000	3.6675
3.2847	0.2417	116000	3.6852
3.2496	0.2437	117000	3.6813
3.281	0.2458	118000	3.6718
3.2697	0.2479	119000	3.6719
3.2208	0.25	120000	3.6595
3.2665	0.2521	121000	3.6706
3.265	0.2542	122000	3.6546
3.3039	0.2562	123000	3.6668
3.2742	0.2583	124000	3.6628
3.2607	0.2604	125000	3.6577
3.2918	0.2625	126000	3.6540
3.2361	0.2646	127000	3.6403
3.2914	0.2667	128000	3.6429
3.2049	0.2687	129000	3.6321
3.2311	0.2708	130000	3.6408
3.2329	0.2729	131000	3.6315
3.2318	0.275	132000	3.6359
3.2287	0.2771	133000	3.6353
3.2889	0.2792	134000	3.6314
3.262	0.2812	135000	3.6412
3.2409	0.2833	136000	3.6325
3.2604	0.2854	137000	3.6292
3.1974	0.2875	138000	3.6217
3.2136	0.2896	139000	3.6326
3.2064	0.2917	140000	3.6171
3.2019	0.2938	141000	3.6307
3.2675	0.2958	142000	3.6220
3.2218	0.2979	143000	3.6384
3.232	0.3	144000	3.6349
3.2268	0.3021	145000	3.6209
3.2527	0.3042	146000	3.6154
3.3142	0.3063	147000	3.6200
3.2486	0.3083	148000	3.6195
3.1876	0.3104	149000	3.6249
3.2235	0.3125	150000	3.6284
3.2296	0.3146	151000	3.6220
3.2066	0.3167	152000	3.6319
3.2517	0.3187	153000	3.6236
3.1939	0.3208	154000	3.6191
3.2645	0.3229	155000	3.6178
3.2388	0.325	156000	3.6212
3.1929	0.3271	157000	3.6181
3.1655	0.3292	158000	3.6202
3.1706	0.3312	159000	3.6174
3.2138	0.3333	160000	3.6020
3.2086	0.3354	161000	3.6076
3.2253	0.3375	162000	3.6035
3.2249	0.3396	163000	3.5919
3.1773	0.3417	164000	3.5994
3.1898	0.3438	165000	3.5999
3.2367	0.3458	166000	3.5893
3.1909	0.3479	167000	3.6001
3.158	0.35	168000	3.6071
3.2281	0.3521	169000	3.5893
3.2258	0.3542	170000	3.5900
3.1549	0.3563	171000	3.5793
3.1841	0.3583	172000	3.5745
3.1991	0.3604	173000	3.5817
3.1658	0.3625	174000	3.5939
3.1957	0.3646	175000	3.5897
3.1929	0.3667	176000	3.5836
3.1652	0.3688	177000	3.5806
3.2392	0.3708	178000	3.5756
3.2093	0.3729	179000	3.5837
3.1547	0.375	180000	3.5854
3.2009	0.3771	181000	3.5779
3.2153	0.3792	182000	3.5743
3.2005	0.3812	183000	3.5732
3.1961	0.3833	184000	3.5855
3.1965	0.3854	185000	3.5874
3.143	0.3875	186000	3.5737
3.1589	0.3896	187000	3.5757
3.1506	0.3917	188000	3.5729
3.1764	0.3937	189000	3.5752
3.1561	0.3958	190000	3.5613
3.2287	0.3979	191000	3.5737
3.1566	0.4	192000	3.5636
3.2028	0.4021	193000	3.5612
3.1997	0.4042	194000	3.5643
3.1413	0.4062	195000	3.5683
3.1797	0.4083	196000	3.5553
3.2138	0.4104	197000	3.5717
3.207	0.4125	198000	3.5605
3.1739	0.4146	199000	3.5592
3.1325	0.4167	200000	3.5606
3.1089	0.4188	201000	3.5621
3.1753	0.4208	202000	3.5534
3.1524	0.4229	203000	3.5476
3.1589	0.425	204000	3.5586
3.1351	0.4271	205000	3.5490
3.1081	0.4292	206000	3.5423
3.0467	0.4313	207000	3.5523
3.1734	0.4333	208000	3.5477
3.1146	0.4354	209000	3.5477
3.151	0.4375	210000	3.5322
3.1045	0.4396	211000	3.5322
3.1233	0.4417	212000	3.5326
3.1608	0.4437	213000	3.5327
3.128	0.4458	214000	3.5458
3.1645	0.4479	215000	3.5342
3.2083	0.45	216000	3.5280
3.0882	0.4521	217000	3.5398
3.1598	0.4542	218000	3.5412
3.1839	0.4562	219000	3.5390
3.1684	0.4583	220000	3.5365
3.1318	0.4604	221000	3.5443
3.1431	0.4625	222000	3.5456
3.0537	0.4646	223000	3.5422
3.1322	0.4667	224000	3.5385
3.1526	0.4688	225000	3.5361
3.0659	0.4708	226000	3.5349
3.13	0.4729	227000	3.5297
3.1159	0.475	228000	3.5287
3.0748	0.4771	229000	3.5267
3.1242	0.4792	230000	3.5354
3.1809	0.4813	231000	3.5289
3.1093	0.4833	232000	3.5316
3.1534	0.4854	233000	3.5325
3.134	0.4875	234000	3.5261
3.1468	0.4896	235000	3.5295
3.1688	0.4917	236000	3.5239
3.1643	0.4938	237000	3.5291
3.1373	0.4958	238000	3.5197
3.1588	0.4979	239000	3.5217
3.0473	0.5	240000	3.5267
3.2146	0.5021	241000	3.5212
3.1697	0.5042	242000	3.5290
3.148	0.5062	243000	3.5268
3.0906	0.5083	244000	3.5178
3.1083	0.5104	245000	3.5126
3.1061	0.5125	246000	3.5130
3.111	0.5146	247000	3.4991
3.1511	0.5167	248000	3.5115
3.1264	0.5188	249000	3.5141
3.2034	0.5208	250000	3.5121
3.06	0.5229	251000	3.5153
3.0718	0.525	252000	3.5144
3.0548	0.5271	253000	3.5069
3.1063	0.5292	254000	3.4987
3.0995	0.5312	255000	3.5105
3.1381	0.5333	256000	3.5025
3.0581	0.5354	257000	3.5044
3.1133	0.5375	258000	3.4979
3.1613	0.5396	259000	3.4933
3.0973	0.5417	260000	3.5024
3.0427	0.5437	261000	3.5095
3.0705	0.5458	262000	3.5020
3.1001	0.5479	263000	3.5021
3.0793	0.55	264000	3.5032
3.1197	0.5521	265000	3.5044
3.0562	0.5542	266000	3.5001
3.1128	0.5563	267000	3.4976
3.073	0.5583	268000	3.5002
3.0071	0.5604	269000	3.4963
3.0537	0.5625	270000	3.5080
3.0515	0.5646	271000	3.5022
2.8561	0.5667	272000	3.4993
2.1328	0.5687	273000	3.5052
1.8604	0.5708	274000	3.5099
1.5633	0.5729	275000	3.5245
1.4802	0.575	276000	3.5352
1.4986	0.5771	277000	3.5362
1.5193	0.5792	278000	3.5498
1.5455	0.5813	279000	3.5489
1.5169	0.5833	280000	3.5541
1.4809	0.5854	281000	3.5662
1.4723	0.5875	282000	3.5740
1.4385	0.5896	283000	3.5717
1.3069	0.5917	284000	3.5847
1.5347	0.5938	285000	3.5832
1.3994	0.5958	286000	3.5954
1.3792	0.5979	287000	3.5923
1.415	0.6	288000	3.6055
1.5194	0.6021	289000	3.6088
1.29	0.6042	290000	3.6132
1.3464	0.6062	291000	3.6106
1.353	0.6083	292000	3.6079
1.3774	0.6104	293000	3.6108
1.4167	0.6125	294000	3.6117
1.2833	0.6146	295000	3.6048
1.344	0.6167	296000	3.6066
1.4293	0.6188	297000	3.6185
1.2332	0.6208	298000	3.6221
1.2435	0.6229	299000	3.6140
1.2532	0.625	300000	3.6197
1.2261	0.6271	301000	3.6293
1.9938	0.6292	302000	3.5898
2.8128	0.6312	303000	3.5598
2.9367	0.6333	304000	3.5419
3.0334	0.6354	305000	3.5330
3.1232	0.6375	306000	3.5259
3.0749	0.6396	307000	3.5236
3.1853	0.6417	308000	3.5134
3.1386	0.6438	309000	3.5126
3.1555	0.6458	310000	3.5069
3.0662	0.6479	311000	3.5027
3.0098	0.65	312000	3.4989
3.0513	0.6521	313000	3.4942
3.081	0.6542	314000	3.4905
3.0592	0.6562	315000	3.4902
3.0992	0.6583	316000	3.4897
3.0761	0.6604	317000	3.4902
3.0232	0.6625	318000	3.4816
3.0426	0.6646	319000	3.4819
3.1092	0.6667	320000	3.4777
3.0312	0.6687	321000	3.4893
3.0898	0.6708	322000	3.4896
3.0576	0.6729	323000	3.4824
2.9931	0.675	324000	3.4772
3.0077	0.6771	325000	3.4817
3.0831	0.6792	326000	3.4760
3.1157	0.6813	327000	3.4756
3.0822	0.6833	328000	3.4752
3.1263	0.6854	329000	3.4697
3.0483	0.6875	330000	3.4760
3.0984	0.6896	331000	3.4763
3.0595	0.6917	332000	3.4730
3.0526	0.6937	333000	3.4710
3.0795	0.6958	334000	3.4736
3.0524	0.6979	335000	3.4734
3.0192	0.7	336000	3.4656
3.0174	0.7021	337000	3.4687
3.0312	0.7042	338000	3.4715
3.0186	0.7063	339000	3.4701
3.0222	0.7083	340000	3.4639
3.0076	0.7104	341000	3.4650
3.0714	0.7125	342000	3.4623
2.9175	0.7146	343000	3.4622
3.0136	0.7167	344000	3.4644
3.0539	0.7188	345000	3.4577
3.0864	0.7208	346000	3.4561
3.0607	0.7229	347000	3.4545
3.0489	0.725	348000	3.4531
3.0048	0.7271	349000	3.4567
3.0989	0.7292	350000	3.4562
3.0388	0.7312	351000	3.4571
3.0766	0.7333	352000	3.4533
3.084	0.7354	353000	3.4575
3.0652	0.7375	354000	3.4512
3.0437	0.7396	355000	3.4551
3.0438	0.7417	356000	3.4570
3.081	0.7438	357000	3.4536
3.0803	0.7458	358000	3.4533
3.0359	0.7479	359000	3.4584
3.0631	0.75	360000	3.4593
3.0193	0.7521	361000	3.4537
3.1205	0.7542	362000	3.4506
3.0604	0.7562	363000	3.4502
3.0792	0.7583	364000	3.4503
2.9729	0.7604	365000	3.4479
3.0601	0.7625	366000	3.4429
3.0352	0.7646	367000	3.4464
3.0392	0.7667	368000	3.4460
3.066	0.7688	369000	3.4478
3.0074	0.7708	370000	3.4499
3.0857	0.7729	371000	3.4460
3.0815	0.775	372000	3.4425
3.0046	0.7771	373000	3.4449
3.0233	0.7792	374000	3.4455
3.0774	0.7812	375000	3.4469
3.0053	0.7833	376000	3.4447
3.0367	0.7854	377000	3.4424
3.0685	0.7875	378000	3.4449
3.0189	0.7896	379000	3.4436
3.0873	0.7917	380000	3.4360
3.0593	0.7937	381000	3.4390
2.9734	0.7958	382000	3.4363
2.9926	0.7979	383000	3.4385
3.0301	0.8	384000	3.4374
2.9467	0.8021	385000	3.4328
2.9794	0.8042	386000	3.4346
3.0649	0.8063	387000	3.4329
3.0733	0.8083	388000	3.4324
3.0386	0.8104	389000	3.4332
3.013	0.8125	390000	3.4350
3.0386	0.8146	391000	3.4324
3.0885	0.8167	392000	3.4341
2.996	0.8187	393000	3.4339
3.1084	0.8208	394000	3.4343
3.0	0.8229	395000	3.4316
2.9807	0.825	396000	3.4270
3.0788	0.8271	397000	3.4272
3.0419	0.8292	398000	3.4302
2.9872	0.8313	399000	3.4274
2.9853	0.8333	400000	3.4311
3.2761	0.8354	401000	3.4221
3.4203	0.8375	402000	3.4176
3.3776	0.8396	403000	3.4127
3.4266	0.8417	404000	3.4108
3.4513	0.8438	405000	3.4067
3.417	0.8458	406000	3.4066
3.371	0.8479	407000	3.4048
3.4803	0.85	408000	3.4022
3.4137	0.8521	409000	3.4028
3.3304	0.8542	410000	3.4032
3.3579	0.8562	411000	3.4039
3.3827	0.8583	412000	3.3999
3.3244	0.8604	413000	3.3995
3.3754	0.8625	414000	3.3995
3.3883	0.8646	415000	3.3993
3.4301	0.8667	416000	3.3979
3.4519	0.8688	417000	3.3980
3.4338	0.8708	418000	3.3968
3.5211	0.8729	419000	3.3981
3.4089	0.875	420000	3.3970
3.376	0.8771	421000	3.3965
3.3782	0.8792	422000	3.3946
3.4154	0.8812	423000	3.3910
3.3131	0.8833	424000	3.3906
3.3433	0.8854	425000	3.3923
3.4678	0.8875	426000	3.3893
3.3821	0.8896	427000	3.3873
3.4541	0.8917	428000	3.3896
3.3757	0.8938	429000	3.3880
3.4463	0.8958	430000	3.3897
3.4266	0.8979	431000	3.3864
3.3054	0.9	432000	3.3854
3.4259	0.9021	433000	3.3856
3.3983	0.9042	434000	3.3869
3.3211	0.9062	435000	3.3866
3.416	0.9083	436000	3.3850
3.3945	0.9104	437000	3.3847
3.4763	0.9125	438000	3.3849
3.3599	0.9146	439000	3.3836
3.4208	0.9167	440000	3.3843
3.4234	0.9187	441000	3.3829
3.3527	0.9208	442000	3.3838
3.3731	0.9229	443000	3.3830
3.3564	0.925	444000	3.3829
3.4002	0.9271	445000	3.3823
3.4151	0.9292	446000	3.3819
3.3721	0.9313	447000	3.3814
3.3909	0.9333	448000	3.3819
3.3552	0.9354	449000	3.3820
3.3308	0.9375	450000	3.3837
3.3509	0.9396	451000	3.3846
3.404	0.9417	452000	3.3846
3.3481	0.9437	453000	3.3842
3.383	0.9458	454000	3.3844
3.3528	0.9479	455000	3.3834
3.3573	0.95	456000	3.3832
3.3597	0.9521	457000	3.3830
3.3821	0.9542	458000	3.3831
3.4154	0.9563	459000	3.3828
3.3741	0.9583	460000	3.3815
3.4097	0.9604	461000	3.3813
3.5173	0.9625	462000	3.3824
3.3884	0.9646	463000	3.3817
3.3439	0.9667	464000	3.3814
3.3791	0.9688	465000	3.3814
3.4821	0.9708	466000	3.3806
3.3939	0.9729	467000	3.3808
3.426	0.975	468000	3.3814
3.3527	0.9771	469000	3.3811
3.3875	0.9792	470000	3.3811
3.3621	0.9812	471000	3.3810
3.406	0.9833	472000	3.3807
3.4422	0.9854	473000	3.3813
3.465	0.9875	474000	3.3815
3.3671	0.9896	475000	3.3809
3.4621	0.9917	476000	3.3815
3.4379	0.9938	477000	3.3810
3.3686	0.9958	478000	3.3813
3.4255	0.9979	479000	3.3811
3.3782	1.0	480000	3.3814

Framework versions

• Transformers 4.50.3
• Pytorch 2.6.0+cu124
• Datasets 2.21.0
• Tokenizers 0.21.4