Keep ssm_ba.weight and ssm_out.weight in BF16?
#1
by dinerburger - opened
These weights are pretty small; seems that we can probably increase SSM accuracy across the board by keeping these in BF16 instead of pushing them down to 5-6 bits.
These weights are pretty small; seems that we can probably increase SSM accuracy across the board by keeping these in BF16 instead of pushing them down to 5-6 bits.