|
---
|
|
language:
|
|
- en
|
|
- de
|
|
- fr
|
|
- it
|
|
- pt
|
|
- hi
|
|
- es
|
|
- th
|
|
license: llama3.1
|
|
pipeline_tag: text-generation
|
|
tags:
|
|
- facebook
|
|
- meta
|
|
- pytorch
|
|
- llama
|
|
- llama-3
|
|
---
|
|
|
|
# Meta-Llama-3.1-70B-Instruct-FP8-128K
|
|
|
|
## Model Overview
|
|
- Model Architecture: Meta-Llama-3.1
|
|
- Input: Text
|
|
- Output: Text
|
|
- Model Optimizations:
|
|
- Weight quantization: FP8
|
|
- Activation quantization: FP8
|
|
- KV Cache quantization:FP8
|
|
- Intended Use Cases: Intended for commercial and research use in multiple languages. Similarly to Meta-Llama-3.1-8B-Instruct, this models is intended for assistant-like chat.
|
|
- Out-of-scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in languages other than English.
|
|
- Release Date: 8/27/2024
|
|
- Version: 1.0
|
|
- License(s): llama3.1
|
|
- Quantized version of Meta-Llama-3.1-8B-Instruct.
|
|
|
|
|
|
## Serve with vLLM engine
|
|
```bash
|
|
python3 -m vllm.entrypoints.openai.api_server \
|
|
--port <port> --model yejingfu/Meta-Llama-3.1-70B-Instruct-FP8-128K \
|
|
--tensor-parallel-size 4 --swap-space 16 --gpu-memory-utilization 0.96 --dtype auto \
|
|
--max-num-seqs 32 --max-model-len 131072 --kv-cache-dtype fp8 --enable-chunked-prefill
|
|
```
|
|
|
|
---
|
|
license: llama3.1
|
|
---
|
|
|
