KLD/PPL of quants
#8
by krampenschiesser - opened
wikitext-2-raw-v1, unsloth q8_0 as base
| Provider | Quant | Size GB | Mean PPL | Mean KLD | Same Top p |
|---|---|---|---|---|---|
| Unsloth | Q8 | 4.3155 +/- 0.02446 | baseline | baseline | |
| Unsloth | UD-Q6_K_XL | 105.0 | 4.317536 ± 0.024475 | 0.004961 ± 0.000192 | 97.655 ± 0.039 % |
| Aes Sedai | Q5_K_M | 91.5 | 4.320741 ± 0.024486 | 0.005936 ± 0.000234 | 97.348 ± 0.042 % |
| Unsloth | Q6_K_M | 101.0 | 4.320079 ± 0.024524 | 0.006602 ± 0.000252 | 97.057 ± 0.044 % |
| Unsloth | Q5_K_M | 87.1 | 4.332594 ± 0.024603 | 0.010502 ± 0.000261 | 96.318 ± 0.049 % |
| Aes Sedai | Q4_K_M | 76.7 | 4.325629 ± 0.024507 | 0.010749 ± 0.000228 | 96.435 ± 0.048 % |
| Unsloth | UD-Q5_K_XL | 87.0 | 4.331663 ± 0.024585 | 0.011109 ± 0.000284 | 96.301 ± 0.049 % |
| Aes Sedai | IQ4_X_S | 60.4 | 4.404998 ± 0.025001 | 0.027409 ± 0.000300 | 94.259 ± 0.060 % |
| Unsloth | Q4K_M | 74.3 | 4.435888 ± 0.025722 | 0.033208 ± 0.000312 | 92.935 ± 0.067 % |
| Unsloth | IQ4_NL | 69.2 | 4.468707 ± 0.026029 | 0.038368 ± 0.000331 | 92.349 ± 0.069 % |
| Unsloth | IQ4_XS | 65.5 | 4.462136 ± 0.025988 | 0.038909 ± 0.000371 | 92.321 ± 0.069 % |
| Unsloth | MXFP4 | 68.3 | 4.452131 ± 0.025427 | 0.057660 ± 0.000527 | 91.221 ± 0.073 % |
| Noctrex | MXFP4 | 74.0 | 4.450555 ± 0.025420 | 0.057950 ± 0.000517 | 91.160 ± 0.074 % |
| Unsloth | IQ3_XXS | 50.5 | 4.567684 ± 0.026154 | 0.068894 ± 0.000561 | 90.486 ± 0.076 % |
| Aes Sedai | IQ3_S | 46.6 | 4.570771 ± 0.026085 | 0.073494 ± 0.000597 | 90.410 ± 0.076 % |
| Unsloth | Q3_K_M | 58.8 | 4.648459 ± 0.027585 | 0.083953 ± 0.000570 | 88.692 ± 0.082 % |
| Unsloth | UD-Q3_K_XL | 54.6 | 4.915599 ± 0.028904 | 0.128848 ± 0.000917 | 87.006 ± 0.087 % |
| Unsloth | UD-Q4_K_XL | 68.4 | 4.867515 ± 0.028856 | 0.130354 ± 0.000939 | 86.819 ± 0.088 % |
| Unsloth | UD-Q2_K_XL | 46.7 | 5.133302 ± 0.030444 | 0.174476 ± 0.001143 | 84.785 ± 0.093 % |
| Aes Sedai | IQ2_XXS | 33.9 | 5.105667 ± 0.030043 | 0.178437 ± 0.001154 | 84.945 ± 0.093 % |
There seems to be something wrong with the Unsloth dynamic 2 quants
Thank you very much for your helpful elaboration.