MohamedAhmedAE/distil_MedGemma_4B_Llama-3.2-1B

Model checkpoint at step 319154 Training config: {'alpha': 0.7, 'temperature': 4, 'accumulation_steps': 1, 'hidden_dim': 1024, 'learning_rate': 1e-06, 'num_epochs': 10, 'teacher_vocab_size': 262208, 'student_vocab_size': 128256, 'save_every_steps': 5000, 'checkpoint_dir': './checkpoints', 'hf_repo_id': 'MohamedAhmedAE/distil_MedGemma_4B_Llama-3.2-1B', 'hf_private': False, 'attention_distill': True, 'feature_matching': True, 'progressive_unfreezing': True, 'adaptive_temperature': True, 'layer_wise_distill': True, 'attention_loss_weight': 0.05, 'feature_loss_weight': 0.1, 'min_temperature': 1.0, 'max_temperature': 8.0, 'unfreeze_schedule': 'linear'}