I use vllm==0.6.3 load this model，it generate the fowllowing error
#1
by
wc-llm
- opened
when I use vllm==0.6.3 load this model，it generate the fowllowing error
File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/parameter.py", line 133, in load_qkv_weight
assert param_data.shape == loaded_weight.shape
AssertionError
torch.Size([6144, 768]) != torch.Size([4608, 768])