Q5_K_S.gguf appears to be broken
#5
by jimmoffet - opened
Q5_K_S is 20% smaller than Q4_K_S for this model. That can't be right, can it?
Q5_K_S.gguf fails to load and gives a value error for the 70B model, using llama.cpp.
Q5_K_S is working fine for me for the 7B and 13B chat models
Q5_K_M also works fine for me for this model, seems like it's only Q5_K_S that's failing.
jimmoffet changed discussion status to closed
jimmoffet changed discussion status to open
Koboldcpp says, "Unknown model, cannot load."
I'm also seeing:
Error: Failed to load model 'TheBloke • llama 2 chat 70B q5_k_s gguf'
This is on LM Studio 0.2.8
I'm also having problem with the file.
gguf_init_from_file: invalid magic characters ''
same error at Mac M3 Pro 36GB
gguf_init_from_file: invalid magic characters ''
Confirmed same issue for me
gguf_init_from_file: invalid magic characters ''