Commit
•
6280b27
1
Parent(s):
693dbd1
Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -1,3 +1,73 @@
|
|
| 1 |
-
---
|
| 2 |
-
|
| 3 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
datasets: wikitext
|
| 3 |
+
license: other
|
| 4 |
+
license_link: https://llama.meta.com/llama3/license/
|
| 5 |
+
---
|
| 6 |
+
This is a quantized model of [Meta-Llama-3-70B-Instruct.yaml](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct.yaml) using GPTQ developed by [IST Austria](https://ist.ac.at/en/research/alistarh-group/)
|
| 7 |
+
using the following configuration:
|
| 8 |
+
- 8bit
|
| 9 |
+
- Act order: True
|
| 10 |
+
- Group size: 128
|
| 11 |
+
|
| 12 |
+
## Usage
|
| 13 |
+
Install **vLLM** and
|
| 14 |
+
run the [server](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#openai-compatible-server):
|
| 15 |
+
|
| 16 |
+
```
|
| 17 |
+
python -m vllm.entrypoints.openai.api_server --model cortecs/Meta-Llama-3-70B-Instruct-GPTQ-8b
|
| 18 |
+
```
|
| 19 |
+
Access the model:
|
| 20 |
+
```
|
| 21 |
+
curl http://localhost:8000/v1/completions -H "Content-Type: application/json" -d ' {
|
| 22 |
+
"model": "cortecs/Meta-Llama-3-70B-Instruct-GPTQ-8b",
|
| 23 |
+
"prompt": "San Francisco is a"
|
| 24 |
+
} '
|
| 25 |
+
```
|
| 26 |
+
|
| 27 |
+
## Evaluations
|
| 28 |
+
| __English__ | __[Meta-Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct)__ | __[Meta-Llama-3-70B-Instruct-GPTQ-8b](https://huggingface.co/cortecs/Meta-Llama-3-70B-Instruct-GPTQ-8b)__ | __[Meta-Llama-3-70B-Instruct-GPTQ](https://huggingface.co/cortecs/Meta-Llama-3-70B-Instruct-GPTQ)__ |
|
| 29 |
+
|:--------------|:-----------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------|
|
| 30 |
+
| Avg. | 76.19 | 76.16 | 75.14 |
|
| 31 |
+
| ARC | 71.6 | 71.4 | 70.7 |
|
| 32 |
+
| Hellaswag | 77.3 | 77.1 | 76.4 |
|
| 33 |
+
| MMLU | 79.66 | 79.98 | 78.33 |
|
| 34 |
+
| | | | |
|
| 35 |
+
| __French__ | __[Meta-Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct)__ | __[Meta-Llama-3-70B-Instruct-GPTQ-8b](https://huggingface.co/cortecs/Meta-Llama-3-70B-Instruct-GPTQ-8b)__ | __[Meta-Llama-3-70B-Instruct-GPTQ](https://huggingface.co/cortecs/Meta-Llama-3-70B-Instruct-GPTQ)__ |
|
| 36 |
+
| Avg. | 70.97 | 71.03 | 70.27 |
|
| 37 |
+
| ARC_fr | 65.0 | 65.3 | 64.7 |
|
| 38 |
+
| Hellaswag_fr | 72.4 | 72.4 | 71.4 |
|
| 39 |
+
| MMLU_fr | 75.5 | 75.4 | 74.7 |
|
| 40 |
+
| | | | |
|
| 41 |
+
| __German__ | __[Meta-Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct)__ | __[Meta-Llama-3-70B-Instruct-GPTQ-8b](https://huggingface.co/cortecs/Meta-Llama-3-70B-Instruct-GPTQ-8b)__ | __[Meta-Llama-3-70B-Instruct-GPTQ](https://huggingface.co/cortecs/Meta-Llama-3-70B-Instruct-GPTQ)__ |
|
| 42 |
+
| Avg. | 68.43 | 68.37 | 66.93 |
|
| 43 |
+
| ARC_de | 64.2 | 64.3 | 62.6 |
|
| 44 |
+
| Hellaswag_de | 67.8 | 67.7 | 66.7 |
|
| 45 |
+
| MMLU_de | 73.3 | 73.1 | 71.5 |
|
| 46 |
+
| | | | |
|
| 47 |
+
| __Italian__ | __[Meta-Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct)__ | __[Meta-Llama-3-70B-Instruct-GPTQ-8b](https://huggingface.co/cortecs/Meta-Llama-3-70B-Instruct-GPTQ-8b)__ | __[Meta-Llama-3-70B-Instruct-GPTQ](https://huggingface.co/cortecs/Meta-Llama-3-70B-Instruct-GPTQ)__ |
|
| 48 |
+
| Avg. | 70.17 | 70.43 | 68.63 |
|
| 49 |
+
| ARC_it | 64.0 | 64.3 | 62.1 |
|
| 50 |
+
| Hellaswag_it | 72.6 | 72.4 | 71.0 |
|
| 51 |
+
| MMLU_it | 73.9 | 74.6 | 72.8 |
|
| 52 |
+
| | | | |
|
| 53 |
+
| __Safety__ | __[Meta-Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct)__ | __[Meta-Llama-3-70B-Instruct-GPTQ-8b](https://huggingface.co/cortecs/Meta-Llama-3-70B-Instruct-GPTQ-8b)__ | __[Meta-Llama-3-70B-Instruct-GPTQ](https://huggingface.co/cortecs/Meta-Llama-3-70B-Instruct-GPTQ)__ |
|
| 54 |
+
| Avg. | 64.28 | 64.17 | 63.64 |
|
| 55 |
+
| RealToxicityPrompts | 97.9 | 97.8 | 98.1 |
|
| 56 |
+
| TruthfulQA | 61.91 | 61.67 | 59.91 |
|
| 57 |
+
| CrowS | 33.04 | 33.04 | 32.92 |
|
| 58 |
+
| | | | |
|
| 59 |
+
| __Spanish__ | __[Meta-Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct)__ | __[Meta-Llama-3-70B-Instruct-GPTQ-8b](https://huggingface.co/cortecs/Meta-Llama-3-70B-Instruct-GPTQ-8b)__ | __[Meta-Llama-3-70B-Instruct-GPTQ](https://huggingface.co/cortecs/Meta-Llama-3-70B-Instruct-GPTQ)__ |
|
| 60 |
+
| Avg. | 72.5 | 72.7 | 71.3 |
|
| 61 |
+
| ARC_es | 66.7 | 66.9 | 65.7 |
|
| 62 |
+
| Hellaswag_es | 75.8 | 75.9 | 74 |
|
| 63 |
+
| MMLU_es | 75 | 75.3 | 74.2 |
|
| 64 |
+
|
| 65 |
+
We did not check for data contamination.
|
| 66 |
+
Evaluation was done using [Eval. Harness](https://github.com/EleutherAI/lm-evaluation-harness) using `limit=1000`.
|
| 67 |
+
|
| 68 |
+
## Performance
|
| 69 |
+
| | requests/s | tokens/s |
|
| 70 |
+
|:------------|-------------:|-----------:|
|
| 71 |
+
| NVIDIA L4x4 | 0.27 | 128.81 |
|
| 72 |
+
| NVIDIA L4x8 | 1.31 | 624.61 |
|
| 73 |
+
Performance measured on [cortecs inference](https://cortecs.ai).
|