Malaysian SFT
Collection
SFT using LoRA and DoRA including reasoning. • 9 items • Updated
GLM-4.5-Air-Malaysian
SFT LoRA zai-org/GLM-4.5-Air on Scicom-intl/Malaysian-Instructions/commit/288b358a57765a735d588f73e5e6c212c81429bd
- MoE LoRA SFT done using FSDP2 Fused MoE.
- Multipacking variable length 16384 context length, with global batch size of 32, so global total tokens is 524288.
- All linear layers with experts, rank 256 with alpha multiply by 2.0 + .
- Liger fused cross entropy.
- 1e-4 learning rate, 50 warmup, 3 epoch only.
+ with the rank of each equal to the total rank divided by the number of active experts, https://thinkingmachines.ai/blog/lora/
We only upload the best model
Source code
Source code at https://github.com/Scicom-AI-Enterprise-Organization/small-ablation/blob/main/malaysian-sft
Acknowledgement
Special thanks to https://www.scitix.ai/ for H100 Node!
- Downloads last month
- 2
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Model tree for Scicom-intl/GLM-4.5-Air-Malaysian
Base model
zai-org/GLM-4.5-Air