Llama2-7B-lora-r-32-generic-step-1200-lr-1e-5-labels_40.0-1

This model is a fine-tuned version of meta-llama/Llama-2-7b-hf on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 2.8683

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 50
• training_steps: 1200
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
5.4841	0.9132	50	4.8936
4.261	1.8265	100	3.8934
3.6676	2.7397	150	3.5533
3.3901	3.6530	200	3.3523
3.2032	4.5662	250	3.2089
3.0317	5.4795	300	3.1105
2.9485	6.3927	350	3.0385
2.8449	7.3059	400	2.9848
2.761	8.2192	450	2.9489
2.7071	9.1324	500	2.9234
2.6587	10.0457	550	2.9012
2.6108	10.9589	600	2.8858
2.5692	11.8721	650	2.8768
2.5821	12.7854	700	2.8691
2.6069	13.6986	750	2.8659
2.5591	14.6119	800	2.8627
2.5155	15.5251	850	2.8598
2.4777	16.4384	900	2.8587
2.5258	17.3516	950	2.8587
2.5114	18.2648	1000	2.8598
2.5042	19.1781	1050	2.8618
2.4948	20.0913	1100	2.8649
2.5022	21.0046	1150	2.8673
2.5069	21.9178	1200	2.8683

Framework versions

• PEFT 0.15.2
• Transformers 4.45.2
• Pytorch 2.5.0+cu121
• Datasets 3.2.0
• Tokenizers 0.20.3