|
--- |
|
license: llama3.3 |
|
--- |
|
|
|
The original [Llama 3.3 70B Instruct](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct) model quantized using AutoAWQ. Follow the instruction [here](https://docs.vllm.ai/en/latest/quantization/auto_awq.html). |
|
|
|
``` |
|
from awq import AutoAWQForCausalLM |
|
from transformers import AutoTokenizer |
|
|
|
model_path = 'meta-llama/Llama-3.3-70B-Instruct' |
|
quant_path = 'Llama-3.3-70B-Instruct-AWQ-4bit' |
|
quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" } |
|
|
|
# Load model |
|
model = AutoAWQForCausalLM.from_pretrained( |
|
model_path, **{"low_cpu_mem_usage": True, "use_cache": False} |
|
) |
|
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True) |
|
|
|
# Quantize |
|
model.quantize(tokenizer, quant_config=quant_config) |
|
|
|
# Save quantized model |
|
model.save_quantized(quant_path) |
|
tokenizer.save_pretrained(quant_path) |
|
``` |
|
|
|
|
|
vLLM serve |
|
``` |
|
vllm serve lambdalabs/Llama-3.3-70B-Instruct-AWQ-4bit \ |
|
--swap-space 16 \ |
|
--disable-log-requests \ |
|
--tokenizer meta-llama/Llama-3.3-70B-Instruct \ |
|
--tensor-parallel-size 2 |
|
``` |
|
|
|
|
|
Benchmark |
|
``` |
|
python benchmark_serving.py \ |
|
--backend vllm \ |
|
--model lambdalabs/Llama-3.3-70B-Instruct-AWQ-4bit \ |
|
--tokenizer meta-llama/Meta-Llama-3-70B \ |
|
--dataset-name sharegpt \ |
|
--dataset-path ./ShareGPT_V3_unfiltered_cleaned_split.json \ |
|
--num-prompts 1000 |
|
|
|
============ Serving Benchmark Result ============ |
|
Successful requests: 902 |
|
Benchmark duration (s): 128.07 |
|
Total input tokens: 177877 |
|
Total generated tokens: 182359 |
|
Request throughput (req/s): 7.04 |
|
Output token throughput (tok/s): 1423.85 |
|
Total Token throughput (tok/s): 2812.71 |
|
---------------Time to First Token---------------- |
|
Mean TTFT (ms): 47225.59 |
|
Median TTFT (ms): 43313.95 |
|
P99 TTFT (ms): 105587.66 |
|
-----Time per Output Token (excl. 1st token)------ |
|
Mean TPOT (ms): 141.01 |
|
Median TPOT (ms): 148.94 |
|
P99 TPOT (ms): 174.16 |
|
---------------Inter-token Latency---------------- |
|
Mean ITL (ms): 131.55 |
|
Median ITL (ms): 150.82 |
|
P99 ITL (ms): 344.50 |
|
================================================== |
|
``` |
