Getting an error in koboldcpp?
Here is an error I get after I used the copy /b command on windows to merge the files. Any idea on how to fix? Thanks.
Hi
Could you show me how you run it?
I started to write a generic answer, before I noticed your "I used the copy /b command on windows to merge the files. ".
I don't think you merge these files with that command.
You use that COPY /B -command only if the split files names end up with something like ".part0" (not like ".gguf").
These files have to be merged with llama.cpp (if you want to merge them).
BUT YOU DON'T HAVE TO MERGE THESE FILES.
You can run these with koboldcpp just by choosing the first file (for example WizardLM-2-8x22B.Q2_K-00001-of-00005.gguf) and put the "GPU layers" down to 0 or other small number.
Below just a generic answer before I noticed your "I used the copy /b command on windows to merge the files. " (I hope these helps others)
Do you have the newest version of koboldcpp? ( koboldcpp-1.62.2)
When trying to load the model, do you choose the first file of the many files? (for example WizardLM-2-8x22B.Q2_K-00001-of-00005.gguf)
When there are model split in many files, koboldcpp automatically sets too many GPU Layers. (Try "GPU Layers 0" or other small number)
Do you get koboldcpp to work with other models? (Maybe the problem is not spesific to this model?)
Many thanks
@AiCreatornator
, I completely missed the copy b part! And thank you for the detailed step by step instructions.
The issue ended up being that I was using the nesenex's fork of kobold as that's usually faster for me, the normal kobold branch works fine. Thank you!