differences between quants
hey...
i would like to ask what are the main differences between
mradermacher
bartowski
unsloth
and yours
what do you think is your advantage ?
any benchmarks available?
Unsloth does extra work on layers to optimize.
bartowski are pretty standard as llama-cpp produces them.
mradermacher the same I think, sometimes also uses custom matrix files.
My quants have extra work to bump certain layers, but its a pretty crude method . I think Unsloth does more work on errors so they might be better sometimes. My approach is not as dynamic, You can see the code here https://github.com/Mungert69/GGUFModelBuilder/blob/main/model-converter/tensor_list_builder.py but you can tweak it all you like with the quant_rules.json . I don't have the resources to do anything more than quantize the smaller models. currently running on old low cost vm. If I had more resources I would do more research and produce more quants.
Overall Unsloth best, My method open source allows you do your own tweaks, Bartowski and mradermacher pretty much standard and the same.
i see ... only curious ;)
you have some python skills , you have time for some topics arround RAG ?
yeah but my knowledge may be limited
i see ... only curious ;)
you have some python skills , you have time for some topics arround RAG ?