metadata
inference: false
weblab-10b-instruction-sft-GPTQ
original model weblab-10b-instruction-sft
This is 4bit GPTQ Version.
The size is smaller and the execution speed is faster, but the inference performance may be a little worse.
sample code
At least one GPU is currently required due to a limitation of the Accelerate library.
So this model cannot be run with the huggingface space free version.
pip install auto-gptq
from transformers import AutoTokenizer
from auto_gptq import AutoGPTQForCausalLM
quantized_model_dir = "dahara1/weblab-10b-instruction-sft-GPTQ"
model_basename = "gptq_model-4bit-128g"
tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir)
model = AutoGPTQForCausalLM.from_quantized(
quantized_model_dir,
model_basename=model_basename,
use_safetensors=True,
device="cuda:0")
prompt = "スタジオジブリの作品を５つ教えてください"
prompt_template = f"### 指示: {prompt}\n\n### 応答:"
tokens = tokenizer(prompt_template, return_tensors="pt").to("cuda:0").input_ids
output = model.generate(input_ids=tokens, max_new_tokens=100, do_sample=True, temperature=0.8)
print(tokenizer.decode(output[0]))
Other documents
https://github.com/PanQiWei/AutoGPTQ/blob/main/docs/tutorial/01-Quick-Start.md
Original Authors
Takeshi Kojima
Benchmark
The results below are preliminary. The blank part is under measurement.
Also, the score may change as a result of tuning after this.
Japanese benchmark
- We used Stability-AI/lm-evaluation-harness + gptq patch for evaluation.
- The 4-task average accuracy is based on results of JCommonsenseQA-1.1, JNLI-1.1, MARC-ja-1.1, and JSQuAD-1.1.
- model loading is performed with gptq_use_triton=True, and evaluation is performed with template version 0.3 using the few-shot in-context learning.
- The number of few-shots is 3,3,3,2.
Model Average JCommonsenseQA JNLI MARC-ja JSQuAD weblab-10b-instruction-sft 78.78 74.35 65.65 96.06 79.04 weblab-10b 66.38 65.86 54.19 84.49 60.98 weblab-10b-instruction-sft-GPTQ - 74.53 41.70 - 72.69