End of training

README.md

@@ -21,7 +21,7 @@ should probably proofread and complete it, then remove this comment. -->
 axolotl version: `0.6.0`
 ```yaml
 base_model: JackFram/llama-68m
-batch_size: 64
+batch_size: 128
 bf16: true
 chat_template: tokenizer_default_fallback_alpaca
 datasets:
@@ -37,7 +37,7 @@ datasets:
     system_prompt: ''
 device_map: auto
 eval_sample_packing: false
-eval_steps: 50
+eval_steps: 200
 flash_attention: true
 gradient_checkpointing: true
 group_by_length: true
@@ -46,7 +46,7 @@ hub_strategy: checkpoint
 learning_rate: 0.0002
 logging_steps: 10
 lr_scheduler: cosine
-max_steps: 5000
+max_steps: 10000
 micro_batch_size: 32
 model_type: AutoModelForCausalLM
 num_epochs: 100
@@ -55,13 +55,15 @@ output_dir: /root/.sn56/axolotl/tmp/llama-68m
 pad_to_sequence_len: true
 resize_token_embeddings_to_32x: false
 sample_packing: true
-save_steps: 50
-save_total_limit: 2
+save_steps: 200
+save_total_limit: 1
 sequence_len: 2048
 special_tokens:
   pad_token: </s>
 tokenizer_type: LlamaTokenizerFast
 torch_dtype: bf16
+training_args_kwargs:
+  hub_private_repo: true
 trust_remote_code: true
 val_set_size: 0.1
 wandb_entity: ''
@@ -79,8 +81,6 @@ warmup_ratio: 0.05
 # llama-68m
 
 This model is a fine-tuned version of [JackFram/llama-68m](https://huggingface.co/JackFram/llama-68m) on the argilla/databricks-dolly-15k-curated-en dataset.
-It achieves the following results on the evaluation set:
-- Loss: 4.0103
 
 ## Model description
 
@@ -104,31 +104,19 @@ The following hyperparameters were used during training:
 - eval_batch_size: 32
 - seed: 42
 - distributed_type: multi-GPU
-- num_devices: 2
-- total_train_batch_size: 64
-- total_eval_batch_size: 64
+- num_devices: 4
+- total_train_batch_size: 128
+- total_eval_batch_size: 128
 - optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
 - lr_scheduler_type: cosine
-- lr_scheduler_warmup_steps: 30
-- training_steps: 600
+- lr_scheduler_warmup_steps: 5
+- training_steps: 100
 
 ### Training results
 
-| Training Loss | Epoch   | Step | Validation Loss |
-|:-------------:|:-------:|:----:|:---------------:|
-| No log        | 0.0769  | 1    | 3.9168          |
-| 2.5978        | 3.8462  | 50   | 2.8149          |
-| 2.0808        | 7.6923  | 100  | 2.9664          |
-| 1.6294        | 11.5385 | 150  | 3.2337          |
-| 1.2699        | 15.3846 | 200  | 3.5217          |
-| 1.0092        | 19.2308 | 250  | 3.7262          |
-| 0.8392        | 23.0769 | 300  | 3.8683          |
-| 0.7428        | 26.9231 | 350  | 3.9435          |
-| 0.6952        | 30.7692 | 400  | 3.9860          |
-| 0.6762        | 34.6154 | 450  | 3.9990          |
-| 0.6739        | 38.4615 | 500  | 4.0167          |
-| 0.6691        | 42.3077 | 550  | 4.0208          |
-| 0.6667        | 46.1538 | 600  | 4.0103          |
+| Training Loss | Epoch  | Step | Validation Loss |
+|:-------------:|:------:|:----:|:---------------:|
+| No log        | 0.1667 | 1    | 3.9323          |
 
 
 ### Framework versions