|
--- |
|
license: cc-by-nc-4.0 |
|
language: |
|
- en |
|
pipeline_tag: text-generation |
|
tags: |
|
- text-generation-inference |
|
- meta |
|
- llama |
|
- llama-3.1 |
|
- autoawq |
|
- awq |
|
--- |
|
|
|
## If You Would Like A Slightly More Robust Version Of This Model Please Check It Out [Here!](https://huggingface.co/UCLA-EMC/Meta-Llama-3.1-8B-Instruct-AWQ-INT4-32-2.17B) |
|
|
|
# Model Information |
|
|
|
The Meta Llama 3.1 collection of multilingual large language models (LLMs) is a collection of pretrained and instruction tuned generative models in 8B, 70B and 405B sizes (text in/text out). The Llama 3.1 instruction tuned text only models (8B, 70B, 405B) are optimized for multilingual dialogue use cases and outperform many of the available open source and closed chat models on common industry benchmarks. |
|
|
|
# Quantized Model |
|
|
|
This is the quantizied version of the BASE model of LLama-3.1-8B. This repo contains the meta-llama/Meta-Llama-3.1-8B quantized down to INT4 with AutoAWQ using GEMM kernels performing zero-point quantization with a group size of 128. |
|
|
|
# Usage Requirements |
|
|
|
The model inference requires 7.29 GB of VRAM to load the model checkpoint. This model has been tested to run on a T4 GPU but can be run on a slightly less powerful machine if needed. |
|
|
|
# Running The Model |
|
|
|
Please be sure to install the following packages before running the sample code. |
|
|
|
```terminal |
|
pip install -q --upgrade transformers accelerate |
|
pip install autoawq |
|
``` |
|
Below is an example of how this model can be run. Simply replace the 'prompt' with an input of your choosing and adjust the maximum token size as needed. |
|
|
|
```python |
|
import torch |
|
from transformers import AutoTokenizer, AwqConfig, AutoModelForCausalLM, pipeline |
|
from huggingface_hub import login |
|
|
|
#This code below is if you are using a hugging face token on google colab. |
|
from google.colab import userdata |
|
my_token = userdata.get("HF_TOKEN") |
|
login(my_token) |
|
|
|
model_name = "UCLA-EMC/Meta-Llama-3.1-8B-AWQ-INT4" |
|
|
|
quantization_config = AwqConfig( |
|
bits=4, |
|
fuse_max_seq_len=512, # Note: Update this as per your use-case |
|
do_fuse=True, |
|
) |
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id) |
|
model = AutoModelForCausalLM.from_pretrained( |
|
model_name, |
|
torch_dtype=torch.float16, |
|
low_cpu_mem_usage=True, |
|
device_map="auto", |
|
quantization_config=quantization_config |
|
) |
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id) |
|
model = AutoModelForCausalLM.from_pretrained( |
|
model_name, |
|
torch_dtype=torch.float16, |
|
low_cpu_mem_usage=True, |
|
device_map="auto", |
|
quantization_config=quantization_config |
|
) |
|
|
|
text_generator = pipeline( |
|
'text-generation', |
|
model = model, |
|
tokenizer = tokenizer, |
|
max_new_tokens=456, |
|
) |
|
|
|
def get_response(prompt): |
|
response = text_generator(prompt) |
|
gen_text = response[0]['generated_text'] |
|
return gen_text |
|
|
|
prompt = "generate python code to make a bar graph" |
|
llama_response = get_response(prompt) |
|
print(llama_response) |
|
``` |
|
|
|
# Final Statements |
|
|
|
I hope this model proves useful to you in your projects. Good luck and happy coding! |
|
|
|
### Acknowledgements |
|
|
|
The example above uses code from the following sources: |
|
|
|
- hugging-quants/Meta-Llama-3.1-405B-Instruct-AWQ-INT4 |
|
- Author: hugging-quants |
|
- Source: https://huggingface.co/hugging-quants/Meta-Llama-3.1-405B-Instruct-AWQ-INT4 |
|
- License: Llama 3.1 Community License Agreement |
