Need more info about your quant
#1
by shambler74 - opened
We're learning more everyday, about how several settings in LLM_Compressor substantially change how a model behaves.
1). What is the group size you used?
2). How many Calibration Samples did you use?
3). What was your Sequence Length?
I would suggest sharing this in the future with any NVFP4's you provide, as there's a substantial difference when Calib Samples are too small, Seq length is too short, and Group size is too large, etc.
Everything I've quanted has been 256 samples at 4096 length. I also always include the dataset used in the model card.