TheBloke

/

StableBeluga2-70B-GPTQ

like 90

Whenever I try to load it in TGI I end up with this error, llama2 managed to load without problems
https://github.com/huggingface/text-generation-inference

text-generation-inference    |   File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/weights.py", line 129, in <listcomp>
text-generation-inference    |     w = [self.get_tensor(f"{p}.g_idx") for p in prefixes]
text-generation-inference    |   File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/weights.py", line 66, in get_tensor
text-generation-inference    |     filename, tensor_name = self.get_filename(tensor_name)
text-generation-inference    |   File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/weights.py", line 53, in get_filename
text-generation-inference    |     raise RuntimeError(f"weight {tensor_name} does not exist")
text-generation-inference    | RuntimeError: weight model.layers.0.self_attn.q_proj.g_idx does not exist
text-generation-inference    |