pavan01729/llama-8B-medical-alpaca

See axolotl config

axolotl version: 0.6.0

base_model: meta-llama/Meta-Llama-3.1-8B
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
load_in_8bit: false
load_in_4bit: true
strict: false
datasets:
- path: tatsu-lab/alpaca
  type: alpaca
  format: csv
  prompt_template: '### Instruction: {instruction}

    ### Input: {input}

    ### Response: {output}'
dataset_prepared_path: null
val_set_size: 0.1
output_dir: /root/outputs/fine_tuned_model
adapter: qlora
lora_model_dir: null
sequence_len: 2048
sample_packing: true
eval_sample_packing: false
pad_to_sequence_len: true
lora_r: 16
lora_alpha: 8
lora_dropout: 0.05
lora_target_modules: null
lora_target_linear: true
lora_fan_in_fan_out: null
wandb_project: null
wandb_entity: null
wandb_watch: null
wandb_name: null
wandb_log_model: null
gradient_accumulation_steps: 4
micro_batch_size: 1
num_epochs: 10
max_steps: 10000000
optimizer: paged_adamw_32bit
lr_scheduler: cosine
learning_rate: 0.0002
train_on_inputs: false
group_by_length: false
bf16: auto
fp16: null
tf32: false
gradient_checkpointing: true
early_stopping_patience: 3
save_strategy: steps
save_steps: 20
evaluation_strategy: steps
eval_steps: 20
load_best_model_at_end: true
save_total_limit: 3
metric_for_best_model: loss
greater_is_better: false
resume_from_checkpoint: null
local_rank: null
logging_steps: 1
xformers_attention: null
flash_attention: true
warmup_steps: 10
debug: null
deepspeed: null
weight_decay: 0.0
fsdp: null
fsdp_config: null
special_tokens:
  pad_token: <|end_of_text|>
mlflow_tracking_uri: https://mlflow-dev.qpiai-pro.tech
mlflow_experiment_name: llama-8B-medical-alpaca
hf_mlflow_log_artifacts: 'true'
local_files_only: true

root/outputs/fine_tuned_model

This model was trained from scratch on the tatsu-lab/alpaca dataset. It achieves the following results on the evaluation set:

• Loss: 2.0808

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 4
• optimizer: Use OptimizerNames.PAGED_ADAMW with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• training_steps: 5950

Training results

Training Loss	Epoch	Step	Validation Loss
2.5158	0.0017	1	2.6621
2.326	0.0335	20	2.2586
2.0957	0.0671	40	2.2004
2.0924	0.1006	60	2.1796
2.1954	0.1341	80	2.1625
2.1584	0.1676	100	2.1508
2.2213	0.2012	120	2.1299
2.0102	0.2347	140	2.1306
2.1419	0.2682	160	2.1169
1.8357	0.3018	180	2.1133
2.0238	0.3353	200	2.1090
2.0338	0.3688	220	2.1089
2.0982	0.4023	240	2.0969
2.0284	0.4359	260	2.0978
2.0016	0.4694	280	2.0961
2.0652	0.5029	300	2.0866
2.0064	0.5365	320	2.0939
2.1175	0.5700	340	2.0795
1.943	0.6035	360	2.0803
2.0691	0.6370	380	2.0861
1.8928	0.6706	400	2.0775
2.0693	0.7041	420	2.0903
2.2198	0.7376	440	2.0779
1.7801	0.7712	460	2.0808

Framework versions

• PEFT 0.14.0
• Transformers 4.47.0
• Pytorch 2.3.1+cu121
• Datasets 3.1.0
• Tokenizers 0.21.0