TheBloke

/

vicuna-7B-1.1-GPTQ

like 58

This is one of my older releases, back when I used to release two files in the same repo.

Delete the file called latest.act-order.safetensors and just use the compat.no-act-order file.

Let me know how that goes.

Youre right!, I had those 2 mixed up. been hopping true so many repositories that I had things mixed up.
Thanks you!

Hey Bloke, I tried with both 4 bit quantised 7B and 13B .safetensors models.
The final output looks gibberish. Can u pls let me know what am i missing in the below code?

' ' '
from transformers import AutoTokenizer, pipeline, logging
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
import argparse

quantized_model_dir = "/content/drive/MyDrive/Vicuna/FastChat/models/TheBloke_vicuna-7B-1.1-GPTQ-4bit-128g_actorder"
model_basename = "/content/drive/MyDrive/Vicuna/FastChat/models/TheBloke_vicuna-7B-1.1-GPTQ-4bit-128g_actorder/vicuna-7B-1.1-GPTQ-4bit-128g"

use_triton = False
tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir, use_fast=True)
quantize_config = BaseQuantizeConfig(
bits=4,
group_size=128,
desc_act=False
)

model = AutoGPTQForCausalLM.from_quantized( quantized_model_dir,
use_safetensors=True,
model_basename=model_basename,
device="cuda:0",
use_triton=use_triton,
quantize_config=quantize_config
)

prompt = """ """

inputs = tokenizer(prompt, return_tensors='pt').to('cuda')
tokens = model.generate(
**inputs,
max_new_tokens=2000,
do_sample=True,
temperature=1.0,
top_p=1.0,

truncation=True
)
print(tokenizer.decode(tokens[0], skip_special_tokens=True))

' ' '

Code looks ok. There was a bug in AutoGPTQ 0.3.0 that could cause gibberish output. Please try updating to 0.3.2 with the following:

pip3 uninstall -y auto-gptq
git clone https://github.com/PanQiWei/AutoGPTQ
cd AutoGPTQ
pip3 install -v .

Then test again.