Model Card Details
This is a recipe of int4 model with group_size 128 for meta-llama/Meta-Llama-3.1-8B-Instruct generated by intel/auto-round.
Reproduce the model
This is an outdated recipe. We recommend using symmetric quantization by removing '--asym'
auto-round \
--model_name meta-llama/Meta-Llama-3.1-8B-Instruct \
--device 0 \
--group_size 128 \
--bits 4 \
--nsamples 512 \
--iters 1000 \
--asym \
--model_dtype "fp16" \
--deployment_device 'auto_round' \
--eval_bs 16 \
--output_dir "./tmp_autoround" \
Inference on CPU/HPU//CUDA
1 git clone https://github.com/intel/auto-round && cd auto-round && pip install -vvv --no-build-isolation -e .
2 CPU only machine: pip install intel-extension-for-transformers
HPU: docker image with Gaudi Software Stack is recommended, please refer to following script for environment setup. More details can be found in Gaudi Guide.
CUDA: no extra operations
from transformers import AutoModelForCausalLM, AutoTokenizer
from auto_round.auto_quantizer import AutoHfQuantizer
quantized_model_dir = "Intel/Meta-Llama-3.1-8B-Instruct-AutoRound"
tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir)
model = AutoModelForCausalLM.from_pretrained(quantized_model_dir,
device_map="auto"
)
##import habana_frameworks.torch.core as htcore ## uncommnet it for HPU
##import habana_frameworks.torch.hpu as hthpu ## uncommnet it for HPU
##model = model.to(torch.bfloat16).to("hpu") ## uncommnet it for HPU
tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir)
text = "There is a girl who likes adventure,"
inputs = tokenizer(text, return_tensors="pt").to(model.device)
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))
text = "There is a girl who likes adventure,"
##There is a girl who likes adventure, and she is always ready to go on a journey. She is a traveler, a wanderer, and a seeker of new experiences. She is always looking for something new and exciting, and she is not afraid to take risks. She is a free
text = "Once upon a time, "
##Once upon a time, in the land of the free and the home of the brave, there was a young girl named Lily. She was a bright and curious child, with a heart full of wonder and a mind full of questions. She lived in a small town surrounded by
text = "Which one is bigger, 9.11 or 9.8 "
##Which one is bigger, 9.11 or 9.8 9.11 is bigger. 9.11 is bigger than 9.8. 9.11 is bigger than 9.8. 9.11 is bigger than 9.8. 9.11 is bigger than
text = "Tell me something about Intel. "
##Tell me something about Intel. I've heard they're a big company, but I don't know much about them. What do they do?" \n
##Intel is a multinational technology company that designs and manufactures semiconductor chips, primarily for the computing and communications industries. They are one of
Evaluate the model
lm-eval 0.4.2 eval_bs 16, 0-shot in default
| Metric | BF16 | INT4 | INT4 lmhead |
|---|---|---|---|
| Avg. | 0.6539 | 0.6483 | 0.6454 |
| Avg w/o gsm8k | 0.6425 | 0.6394 | 0.6357 |
| mmlu | 0.6778 | 0.6676 | 0.6602 |
| lambada_openai | 0.7306 | 0.7262 | 0.7180 |
| hellaswag | 0.5911 | 0.5848 | 0.5840 |
| winogrande | 0.7388 | 0.7261 | 0.7198 |
| piqa | 0.7998 | 0.7976 | 0.7992 |
| truthfulqa_mc1 | 0.3709 | 0.3635 | 0.3709 |
| openbookqa | 0.3380 | 0.3440 | 0.3380 |
| boolq | 0.8410 | 0.8361 | 0.8339 |
| arc_easy | 0.8203 | 0.8220 | 0.8182 |
| arc_challenge | 0.5162 | 0.5256 | 0.5145 |
| gsm8k(5shot) strict match | 0.7688 | 0.7377 | 0.7422 |
Ethical Considerations and Limitations
The model can produce factually incorrect output, and should not be relied on to produce factually accurate information. Because of the limitations of the pretrained model and the finetuning datasets, it is possible that this model could generate lewd, biased or otherwise offensive outputs.
Therefore, before deploying any applications of the model, developers should perform safety testing.
Caveats and Recommendations
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.
Here are a couple of useful links to learn more about Intel's AI software:
Disclaimer
The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please consult an attorney before using this model for commercial purposes.