See axolotl config

axolotl version: 0.4.1

adapter: lora
base_model: echarlaix/tiny-random-PhiForCausalLM
bf16: true
chat_template: llama3
dataset_prepared_path: null
datasets:
- data_files:
  - 7462b07f6259b24d_train_data.json
  ds_type: json
  format: custom
  path: /workspace/input_data/7462b07f6259b24d_train_data.json
  type:
    field_instruction: startphrase
    field_output: gold-ending
    format: '{instruction}'
    no_input_format: '{instruction}'
    system_format: '{system}'
    system_prompt: ''
debug: null
device_map:
  ? ''
  : 0,1,2,3,4,5,6,7
early_stopping_patience: 2
eval_max_new_tokens: 128
eval_steps: 400
eval_table_size: null
flash_attention: false
gradient_accumulation_steps: 4
gradient_checkpointing: true
group_by_length: false
hub_model_id: Alphatao/eef53dbf-858f-4886-a97d-23eea0896508
hub_repo: null
hub_strategy: null
hub_token: null
learning_rate: 0.0002
load_best_model_at_end: true
load_in_4bit: false
load_in_8bit: false
local_rank: null
logging_steps: 1
lora_alpha: 32
lora_dropout: 0.05
lora_fan_in_fan_out: null
lora_model_dir: null
lora_r: 16
lora_target_linear: true
lora_target_modules:
- q_proj
- k_proj
- v_proj
- o_proj
lr_scheduler: cosine
max_grad_norm: 1.0
max_steps: 252982
micro_batch_size: 2
mlflow_experiment_name: /tmp/7462b07f6259b24d_train_data.json
model_type: AutoModelForCausalLM
num_epochs: 2
optimizer: adamw_bnb_8bit
output_dir: miner_id_24
pad_to_sequence_len: true
resume_from_checkpoint: null
s2_attention: null
sample_packing: false
save_steps: 400
sequence_len: 1024
special_tokens:
  pad_token: <|endoftext|>
strict: false
tf32: true
tokenizer_type: AutoTokenizer
train_on_inputs: false
trust_remote_code: true
val_set_size: 0.05
wandb_entity: null
wandb_mode: online
wandb_name: 9611c628-3f80-4127-8fd5-47e5a88912ed
wandb_project: Gradients-On-Demand
wandb_run: your_name
wandb_runid: 9611c628-3f80-4127-8fd5-47e5a88912ed
warmup_steps: 10
weight_decay: 0.0
xformers_attention: null

eef53dbf-858f-4886-a97d-23eea0896508

This model is a fine-tuned version of echarlaix/tiny-random-PhiForCausalLM on the None dataset. It achieves the following results on the evaluation set:

• Loss: 6.7601

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 8
• optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• training_steps: 22029

Training results

Training Loss	Epoch	Step	Validation Loss
6.9286	0.0001	1	6.9355
6.8414	0.0363	400	6.8243
6.7953	0.0726	800	6.8112
6.8143	0.1089	1200	6.8057
6.7998	0.1453	1600	6.8016
6.7968	0.1816	2000	6.7978
6.7998	0.2179	2400	6.7946
6.7852	0.2542	2800	6.7911
6.7834	0.2905	3200	6.7886
6.7788	0.3268	3600	6.7859
6.7774	0.3632	4000	6.7832
6.789	0.3995	4400	6.7812
6.8127	0.4358	4800	6.7789
6.8022	0.4721	5200	6.7768
6.7798	0.5084	5600	6.7755
6.7759	0.5447	6000	6.7742
6.7801	0.5811	6400	6.7727
6.7578	0.6174	6800	6.7717
6.8055	0.6537	7200	6.7707
6.8098	0.6900	7600	6.7698
6.771	0.7263	8000	6.7691
6.7894	0.7626	8400	6.7684
6.8033	0.7990	8800	6.7675
6.7812	0.8353	9200	6.7670
6.7753	0.8716	9600	6.7663
6.7672	0.9079	10000	6.7663
6.7683	0.9442	10400	6.7651
6.7629	0.9805	10800	6.7646
6.8388	1.0169	11200	6.7642
6.0088	1.0532	11600	6.7638
7.0827	1.0895	12000	6.7634
6.0642	1.1258	12400	6.7631
7.2639	1.1621	12800	6.7628
6.5203	1.1984	13200	6.7623
6.7918	1.2348	13600	6.7621
7.3091	1.2711	14000	6.7619
6.4894	1.3074	14400	6.7616
7.5799	1.3437	14800	6.7614
5.9648	1.3800	15200	6.7613
6.1966	1.4163	15600	6.7610
6.7871	1.4527	16000	6.7609
6.3081	1.4890	16400	6.7608
6.238	1.5253	16800	6.7607
7.1233	1.5616	17200	6.7606
7.8204	1.5979	17600	6.7606
7.0646	1.6342	18000	6.7605
7.6328	1.6706	18400	6.7604
7.9489	1.7069	18800	6.7603
6.4592	1.7432	19200	6.7602
6.1029	1.7795	19600	6.7602
6.6503	1.8158	20000	6.7602
7.6403	1.8521	20400	6.7601
6.7675	1.8885	20800	6.7601
7.3046	1.9248	21200	6.7601
7.9237	1.9611	21600	6.7601
6.2206	1.9974	22000	6.7601

Framework versions

• PEFT 0.13.2
• Transformers 4.46.0
• Pytorch 2.5.0+cu124
• Datasets 3.0.1
• Tokenizers 0.20.1