AliceThirty

/

GLM-4.7-PRISM-Unsloth-GGUF

like 8

Sure, don't hesitate to ask for a specific quantization, but I gave up on quantizing every UD model because I have to download them all to see how they are quantized.
I tried it and it works, but during reasoning it gets stuck on infinite loops... lol. TQ1_0 is way too much quantized

I have run the Unsloth quantized glm4.7 UD-TQ1_0 model and it has always been able to respond normally without getting stuck in repetitive looping answers

I'm using the non-reasoning mode.

I haven't tried the non-reasoning mode, let me know if it works for you. If it doesn't work, it just means that the imatrix of glm-4.7 is not compatible with glm-4.7-prism

I haven't tried the non-reasoning mode, let me know if it works for you. If it doesn't work, it just means that the imatrix of glm-4.7 is not compatible with glm-4.7-prism

"Non-reasoning mode can respond normally

Has anyone had any luck getting this to work with thinking?

If not, AliceThirsty, is there any chance of a IQ2_XXS iMat quant (not an Unsloth quant, just the normal one?). Maybe the different quant style and size will work better). It's usually a few GB bigger (~88GB) and it works well with the GLM 4.7 model on my 96GB Mac (mind you the UD TQ1_0 quant does too). That's the largest I can normally manage with the GLM quants.

Thanks in advance!

I will work on the UD-Q2_K_L, UD-IQ2_XXS, and IQ2_XXS versions right now

Thank you! I really appreciate it. So you're making an Unsloth and a normal IQ2 XXS quant? They're very different in size (something that always confuses me when trying to compare like for like etc.)

Yes I'll make both to compare them

The Q2_K_XL is uploaded. The UD-IQ2_XXS and IQ2_XXS are ready, but I can't upload them because huggingface generates an error every time, even through the browser

It's the latest huggingface_hub python library by the way, so it should work

Thanks for making them. Have you had a chance to test them to see if either or both have the looping thinking issue or are they okay?

I tried all three of them and none of them are stuck in a loop. Their thought processes are coherent. I really want to upload the IQ2_XXS and UD-IQ2_XSS, but it still doesn't work. I reached out huggingface_hub on github and I'm waiting for someone's help