TypeError: mixtral isn't supported yet.
TypeError: mixtral isn't supported yet.
At:
/opt/conda/envs/condaenv/lib/python3.8/site-packages/auto_gptq/modeling/_utils.py(232): check_and_get_model_type
/opt/conda/envs/condaenv/lib/python3.8/site-packages/auto_gptq/modeling/auto.py(98): from_quantized
/app/mixtral-8x7B-v0.1-GPTQ/1/model.py(56): initialize
I followed instructions from https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GPTQ#python-code-example-inference-from-this-gptq-model (I tried installations from wheel and from source).
Is mixtral not runnable with autoGPTQ yet?
Only mistral's name is supported, you need to change the code https://github.com/PanQiWei/AutoGPTQ/blob/main/auto_gptq/modeling/_const.py
or maybe rename to mistral?
Thanks @Yhyu13 , I renamed mixtral to mistral in config.json and its working but now I'm getting
CUDA extension not installed.
CUDA extension not installed.
CUDA kernels for auto_gptq are not installed, this will result in very slow inference speed. This may because:
1. You disabled CUDA extensions compilation by setting BUILD_CUDA_EXT=0 when install auto_gptq from source.
2. You are using pytorch without CUDA support.
3. CUDA and nvcc are not installed in your device.
I have CUDA installed