"Typo" Fixation and Consistency Issues
#15
by DrBoomstick9000 - opened
I've been running this model on my local setup using llamacpp (2x Tesla P40s). I've tried two different quants (Q4_K_M and Q6) and both seem to run into this issue. Whenever the model "audits" itself during a thinking loop, it detects non existent typos and begins to fixate on them. I even added prompting telling the model to use shell commands to check spellling, and it still seemed to be convinced of non existent typos. Really weird behavior- especially with temp set to 0. Anyone experiencing anything similar?
I just tested the 26B model and it doesn't seem to have the same issue. Very strange, let me know if anyone is experiencing this! Wondering if it is llama cpp specific