---
datasets: wikitext
license: apache-2.0
license_link: https://llama.meta.com/llama3/license/
---
This is a quantized model of [Llama-3 70B Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct) using GPTQ developed by [IST Austria](https://ist.ac.at/en/research/alistarh-group/)
 using the following configuration:
 - 4bit (8bit will follow)
- Act order: True
 - Group size: 128
 - Seq. length: 4096

## Usage
Install **vLLM** and 
    run the [server](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#openai-compatible-server):
    
```
python -m vllm.entrypoints.openai.api_server --model cortecs/Meta-Llama-3-70B-Instruct-GPTQ
```
Access the model:
```
curl http://localhost:8000/v1/completions 
    -H "Content-Type: application/json"
    -d '{
        "model": "cortecs/Meta-Llama-3-70B-Instruct-GPTQ",
        "prompt": "<|begin_of_text|><|start_header_id|>user<|end_header_id|>
Tell me a joke<|eot_id|><|start_header_id|>assistant<|end_header_id|>"
    }'
```

## Evaluations
| __English__   | __Llama-3 70B Instruct__   | __Llama 3 70B GPTQ__   | __Llama-3 8B Instruct__   |
|:--------------|:---------------------------|:-----------------------|:--------------------------|
| Avg.          | 76.19                      | 75.14                  | 66.97                     |
| ARC           | 71.6                       | 70.7                   | 62.5                      |
| Hellaswag     | 77.3                       | 76.4                   | 70.3                      |
| MMLU          | 79.66                      | 78.33                  | 68.11                     |
|               |                            |                        |                           |
| __French__   | __Llama-3 70B Instruct__   | __Llama 3 70B GPTQ__   | __Llama-3 8B Instruct__   |
| Avg.         | 70.97                      | 70.27                  | 57.73                     |
| ARC_fr       | 65.0                       | 64.7                   | 53.3                      |
| Hellaswag_fr | 72.4                       | 71.4                   | 61.7                      |
| MMLU_fr      | 75.5                       | 74.7                   | 58.2                      |
|              |                            |                        |                           |
| __German__   | __Llama-3 70B Instruct__   | __Llama 3 70B GPTQ__   | __Llama-3 8B Instruct__   |
| Avg.         | 68.43                      | 66.93                  | 53.47                     |
| ARC_de       | 64.2                       | 62.6                   | 49.1                      |
| Hellaswag_de | 67.8                       | 66.7                   | 55.0                      |
| MMLU_de      | 73.3                       | 71.5                   | 56.3                      |
|              |                            |                        |                           |
| __Italian__   | __Llama-3 70B Instruct__   | __Llama 3 70B GPTQ__   | __Llama-3 8B Instruct__   |
| Avg.          | 70.17                      | 68.63                  | 56.73                     |
| ARC_it        | 64.0                       | 62.1                   | 51.6                      |
| Hellaswag_it  | 72.6                       | 71.0                   | 61.3                      |
| MMLU_it       | 73.9                       | 72.8                   | 57.3                      |
|               |                            |                        |                           |
| __Safety__          | __Llama-3 70B Instruct__   | __Llama 3 70B GPTQ__   | __Llama-3 8B Instruct__   |
| Avg.                | 64.28                      | 63.64                  | 61.42                     |
| RealToxicityPrompts | 97.9                       | 98.1                   | 97.2                      |
| TruthfulQA          | 61.91                      | 59.91                  | 51.65                     |
| CrowS               | 33.04                      | 32.92                  | 35.42                     |
|                     |                            |                        |                           |
| __Spanish__   |   __Llama-3 70B Instruct__ |   __Llama 3 70B GPTQ__ |   __Llama-3 8B Instruct__ |
| Avg.          |                       72.5 |                   71.3 |                      59   |
| ARC_es        |                       66.7 |                   65.7 |                      54.1 |
| Hellaswag_es  |                       75.8 |                   74   |                      63.8 |
| MMLU_es       |                       75   |                   74.2 |                      59.1 |

Take with caution. We did not check for data contamination.
     Evaluation was done using [Eval. Harness](https://github.com/EleutherAI/lm-evaluation-harness) using `limit=1000` for big datasets. 
    
## Performance
| __Llama-3 70B Instruct__   | __requests/s__   | __tokens/s__   |
|:---------------------------|:-----------------|:---------------|
| NVIDIA L40Sx4              | 2.38             | 1135.41        |
|                            |                  |                |
| __Llama 3 70B GPTQ__   | __requests/s__   | __tokens/s__   |
| NVIDIA L40Sx2          | 2.0              | 951.28         |
|                        |                  |                |
| __Llama-3 8B Instruct__   |   __requests/s__ |   __tokens/s__ |
| NVIDIA L40Sx1             |            11.64 |        5548.63 |
| NVIDIA L4x1               |             2.76 |        1315.25 |
| NVIDIA L4x2               |             4.79 |        2283.53 |
Performance was measured on [cortecs.ai](https://cortecs.ai).