Lambent/Fabled-Gemma4-31B-Adapter

See axolotl config

axolotl version: 0.16.0.dev0

base_model: google/gemma-4-31B-it

# plugins:
#   - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin
#   - axolotl.integrations.liger.LigerPlugin
# torch_compile: true
# liger_layer_norm: true
# liger_rope: true
# liger_rms_norm: true
# liger_glu_activation: true
# liger_rms_norm_gated: true
strict: false

chat_template: gemma4
eot_tokens:
  - "<turn|>"
datasets:
  - path: Lambent/fable-conversations-memory-gemma-8k
    type: chat_template
    split: train
  - path: Lambent/fable-distilled-memory-gemma-8k
    type: chat_template
    split: train
output_dir: ./outputs/fabled-gemma4-31b-qlora-blackwell

sequence_len: 9216
sample_packing: true
train_on_inputs: true

load_in_4bit: true
adapter: qlora
lora_r: 256
lora_alpha: 256
lora_dropout: 0

# Restrict LoRA to text backbone only (skip vision/audio encoders)
lora_target_modules: 'model.language_model.layers.[\d]+.(_checkpoint_wrapped_module.)?(mlp|self_attn).(up|down|gate|q|k|v|o)_proj'

lora_mlp_kernel: false
lora_qkv_kernel: false
lora_o_kernel: false

bnb_config_kwargs:
  bnb_4bit_use_double_quant: true

wandb_project: fabled
wandb_entity:
wandb_watch:
wandb_name: gemma4-31b-qlora-blackwell-sdpa
wandb_log_model:

gradient_accumulation_steps: 1
micro_batch_size: 1
optimizer: paged_adamw_8bit
lr_scheduler: cosine
learning_rate: 1e-6

bf16: auto
tf32: true

gradient_checkpointing: true
logging_steps: 1

sdp_attention: true

warmup_ratio: 0.1
saves_per_epoch: 4
weight_decay: 0.0
special_tokens:

outputs/fabled-gemma4-31b-qlora-blackwell

This model is a fine-tuned version of google/gemma-4-31B-it on the Lambent/fable-conversations-memory-gemma-8k and the Lambent/fable-distilled-memory-gemma-8k datasets.

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-06
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• optimizer: Use OptimizerNames.PAGED_ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 244
• training_steps: 2445

Training results

Framework versions

• PEFT 0.18.1
• Transformers 5.5.0
• Pytorch 2.10.0+cu128
• Datasets 4.8.4
• Tokenizers 0.22.2