llama3.2-3B-added-tokens-wiki-cursor-backspace-left-right-cosine-loss

This model is a fine-tuned version of rocker417/llama3.2-3B-added-tokens-wiki-cursor-backspace-cosine-loss on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.5237

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 4
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 16
• optimizer: Use adamw_8bit with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.03
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss
1.2133	0.0356	1000	1.1457
0.6348	0.0711	2000	0.6532
0.667	0.1067	3000	0.6245
0.5906	0.1422	4000	0.6024
0.5665	0.1778	5000	0.5970
0.5178	0.2133	6000	0.5972
0.5361	0.2489	7000	0.5808
0.5217	0.2844	8000	0.5623
0.5505	0.3200	9000	0.5627
0.5344	0.3556	10000	0.5622
0.5665	0.3911	11000	0.5492
0.5012	0.4267	12000	0.5519
0.4095	0.4622	13000	0.5412
0.5338	0.4978	14000	0.5366
0.5305	0.5333	15000	0.5344
0.5424	0.5689	16000	0.5294
0.506	0.6044	17000	0.5300
0.5096	0.6400	18000	0.5263
0.5467	0.6755	19000	0.5258
0.5197	0.7111	20000	0.5252
0.4869	0.7467	21000	0.5250
0.5439	0.7822	22000	0.5240
0.4992	0.8178	23000	0.5239
0.4298	0.8533	24000	0.5239
0.4991	0.8889	25000	0.5237
0.4843	0.9244	26000	0.5238
0.4968	0.9600	27000	0.5239
0.4853	0.9955	28000	0.5237

Framework versions

• Transformers 4.50.0
• Pytorch 2.3.0+cu118
• Datasets 2.21.0
• Tokenizers 0.21.4