Context Length?
What is the context length for this model?
related, what system prompt format do you recommend?
----- update ------
<|system|>
You are a helpful AI assistant. You answer questions correctly
<|user|>
{{ .Prompt }}
<|assistant|>
The context window can be taken from the llama.cpp output, but without the prompt template we are doomed
According to llama.cpp it's 32k
It's 8k I'll modify the config to reflect that.
Thank you!
related, what system prompt format do you recommend?
----- update ------
<|system|>
You are a helpful AI assistant. You answer questions correctly
<|user|>
{{ .Prompt }}
<|assistant|>
I use chatml by default but this is not supposed to be the right prompt template according to the tokens (Llama's chat template is). It still works so I'm careful about modifying anything (if it ain't broke...).
I think it would be great if you would always publish the context window size and the prompt template in the model card. Thanks.
I think it would be great if you would always publish the context window size and the prompt template in the model card. Thanks.
Yes, it's done. From now on, I'll try to provide this info for every model.
Yes, it's done. From now on, I'll try to provide this info for every model.
Awesome to hear! By the way, can you include the updated metadata in the gguf files themselves? I noticed that the 32k context metadata info messed with the automatic RoPE scaling for KoboldCPP. You are onto something with your model, so it would be much appreciated. Thanks! 🙂
Yes, I'm going to reupload all the GGUF files with the new version