the struggle
For the life of me, i cannot figure out how to use this model without it becoming incoherent after some time.
I've done everything i can think of, i use Danchat template, ive tried using chatml template, setting temp to 1 and top to .95, (with no other settings applied)
tried using dry, shifting other settings, using top k or typical sampling.. and not using them of course, lowering temperature (slight success there, it does make it more coherent BUT STILL INCOHERENT)
ive tried everything i can think of, this model hates me, short of sacrificing my own blood, i dont know what it wants. i KNOW there is a happy place for this model, people have clearly gotten good things from it. i just cant figure it out. ive even tried searching reddit and other places on this and the 24b model, people do have trouble with it. and some people dont. there is a secret sauce, an ingredient that i cannot find somewhere that makes this model happy, i just dont know what it is.
Ive even tried enabling unified kv cache and disabling it, fiddling with those cache settings and whatnot but alas, nothing it seems that i do makes this model happy. i cant tell its a good model, i do get good results.. it just decays after a few messages and forgets how to english.
So what does it want? my first born? a demon ritual of some sorts? should i spin around three times and bark like a dog? i have no clue. it shall continue tormenting me.
Check that the correct template is applied to your prompt. For example, llama.cpp uses a third-party template unless the --jinja flag is explicitly specified. An example of the correct formatting is provided on the model page (remember to ensure the bos token is also present).
incorrect formatting: <|system|>\nsys msg<|user|>\nreq<|assistant|>
correct formatting: <|system|>sys msg<|endoftext|><|user|>req<|endoftext|><|assistant|>
correct with bos: [gMASK]<sop><|system|>sys msg<|endoftext|><|user|>req<|endoftext|><|assistant|>