Qwen 3.5 27b fine tuning multi-gpu example lora sft
#43
by Ethermich - opened
Hi,
Spent lots of time trying to get a fine tuning setup with multi gpu sharding of the model.
However, I could fine tune fine on a single GPU or whith smaller models on a multi gpu setup. It only loaded the full model on each gpu.
Thanks in advance for anyone sharing an example or a tip how to get a working sharding setup for fine tuning with lora or qlora.
Thanks,
Michael