ggufing it
tried gguf-my-repo to have an 8bit gguf for your two instruct models, but it didn't work. I don't get the error message, but I was expecting it to work since it is a llama.
Hi @KnutJaegersberg , I don't know specifically why it's not working as expected, but we will very soon publish quantized versions of our models, so you will be able to use those.
I have quantized and uploaded the models to ollama, they can be run with ease with a single line of code:
ollama run hdnh2006/salamandra-7b-instruct
or
ollama run hdnh2006/salamandra-2b-instruct
I have quantized them from 2 to 8 bits and they can also be downloaded from HuggingFace:
https://huggingface.co/hdnh2006/BSC-LT-salamandra-2b-instruct-gguf
https://huggingface.co/hdnh2006/BSC-LT-salamandra-7b-instruct-gguf
Ollama links:
https://ollama.com/hdnh2006/salamandra-2b-instruct
https://ollama.com/hdnh2006/salamandra-7b-instruct