A favor for the GPU poor?
#1
by BlueNipples - opened
I have compiled textgeneration ui for rocm on my weird amd 8gb vram mobile card, but of course exl3 won't let you quantize without their cuda optimizations.
Any chance you could do a 3bpw of this and help a man out? 🙏
Hey!
Sure, why not :)
Though my internet hasn't been feeling great for the past couple of weeks, so uploading will take a while (maybe a day or two).
Also, I'll upload it to: Crownelius_Crow-9B-HERETIC-4.6_EXL3
Same model, same everything — just with official support by the exllamav3 library (as of version 0.23-ish). The current one is custom support (from before the official release dropped).