MaziyarPanahi

/

Mistral-7B-Instruct-v0.3-GGUF

like 62

Nobody did it, because everyone got an error ... (unsupported/unrecognised model architecture)

Yes, unfortunately those need an update from Llama.cpp first. (it's a shame, they are very good models)

I've updated my llama.cpp to the latest release (and this one recommended by other quantizers). Nonetheless, having the same issue. Unfortunately, I have only Macbook M1 8GB, so this it the only way for me is to run LLM 7-8B, 4q, that's why I'm seeking for any solution.

I hope they fix it soon in the Llama.cpp project

I've updated my llama.cpp to the latest release (and this one recommended by other quantizers). Nonetheless, having the same issue. Unfortunately, I have only Macbook M1 8GB, so this it the only way for me is to run LLM 7-8B, 4q, that's why I'm seeking for any solution.

It's not that easy. Both Phi3 and OpenELM are super hard to wrap your head around for this.