Spaces:
Running
Not following prompts
For some reason system prompts that used to work don't anymore, the AI just doesn't follow it
Maybe you changed models? We rotate models regularly so could be that you're now using a different one ?
@nsarrazin I can confirm this is an issue with multiple models that used to work. None of the 8 models follow the prompt anymore.
Could you share an example, maybe an assistant that doesn't follow its prompt ?
Yup! Here's one I was using with mistralai/Mixtral-8x7B-Instruct-v0.1 :
You're an Islamic research assistant that only uses authentic Islamic texts like the Qur'an and hadith to answer questions.
It used to have an Islamic research assistant persona. For example, it would say Salam Alaykoum when I said hello.
Here's another one that used to work with mistralai/Mixtral-8x7B-Instruct-v0.1 and mistralai/Mistral-7B-Instruct-v0.3:
You're a pirate that only replies in pirate speak.
I used to type hello and it would respond with arrrgghhh...etc
Thanks, I can reproduce the issue, investigating
Awesome, thank you!
It seems to be a mistral issue, for example Llama 3.1 70B seems to work okay. Will check, do you know when this changed approximately ?
It's been at least 3 or 4 days since I noticed. It wasn't until I tested different prompts today and realized none of them worked.
I just tested meta-llama/Meta-Llama-3.1-70B-Instruct and it does seem to work.
The issue seems to only be for mistralai/Mistral-7B-Instruct-v0.3 and mistralai/Mixtral-8x7B-Instruct-v0.1 which is weird since their chat template hasn't been updated for a while so I don't think anything changed...
Foudn the issue! I'll get a fix out for mistralai/Mixtral-8x7B-Instruct-v0.1 but might take a bit longer for mistralai/Mistral-7B-Instruct-v0.3
Somewhat relevant, mistralai/Mistral-7B-Instruct-v0.3 had an update to fix the tool calling, than a follow up update to fix the chat template. Here's the commit in case haven't seen it https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3/commit/b8ef225287903940a8fcc1f6e1016b29c5cc01f3
Yup I'm asking internally about the template, that's why it might take longer 😛
No worries, I was more interested in mistralai/Mixtral-8x7B-Instruct-v0.1 :D
Thank you for looking into it!