Samplers
Nice model.
What is Text completion presets?
Instruct template?
For context it uses ChatML.
ChatML instruct.
For sampler, I recommend temperature and min_p only. Temp around 1-1.1. Min_p around 0.06-0.07
I cannot find settings where the responses aren't nonsense. It seems random, like for a good chunk it's fine responses and then just randomly it will start speaking as if it doesn't speak English and is trying to guess how to talk.
I cannot find settings where the responses aren't nonsense. It seems random, like for a good chunk it's fine responses and then just randomly it will start speaking as if it doesn't speak English and is trying to guess how to talk.
Are you using gguf q4km?
I cannot find settings where the responses aren't nonsense. It seems random, like for a good chunk it's fine responses and then just randomly it will start speaking as if it doesn't speak English and is trying to guess how to talk.
There is a bug with CuBLAS and Qwen2, which this model is based on. I recommend giving a different prompt processing backend like Vulkan a try and seeing if that fixes it