Hugging Face's logo

cortecs
/
Meta-Llama-3-70B-Instruct-GPTQ

Text Generation
Transformers
llama
conversational
Inference Endpoints
4-bit precision
gptq
Model card Files Community
markoarnauto commited on
Commit
2821883
•
1 Parent(s): 83d5513

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +41 -40
README.md CHANGED
@@ -3,9 +3,9 @@ datasets: wikitext
3
  license: other
4
  license_link: https://llama.meta.com/llama3/license/
5
  ---
6
- This is a quantized model of [Llama-3 70B Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct) using GPTQ developed by [IST Austria](https://ist.ac.at/en/research/alistarh-group/)
7
  using the following configuration:
8
- - 4bit (8bit will follow)
9
  - Act order: True
10
  - Group size: 128
11
 
@@ -25,47 +25,48 @@ curl http://localhost:8000/v1/completions -H "Content-Type: application/json
25
  ```
26
 
27
  ## Evaluations
28
- | __English__ | __Llama-3 70B Instruct__ | __Llama 3 70B Instruct GPTQ__ | __Mixtral Instruct__ |
29
- |:--------------|:---------------------------|:--------------------------------|:-----------------------|
30
- | Avg. | 76.19 | 75.14 | 73.17 |
31
- | ARC | 71.6 | 70.7 | 71.0 |
32
- | Hellaswag | 77.3 | 76.4 | 77.0 |
33
- | MMLU | 79.66 | 78.33 | 71.52 |
34
- | | | | |
35
- | __French__ | __Llama-3 70B Instruct__ | __Llama 3 70B Instruct GPTQ__ | __Mixtral Instruct__ |
36
- | Avg. | 70.97 | 70.27 | 68.7 |
37
- | ARC_fr | 65.0 | 64.7 | 63.9 |
38
- | Hellaswag_fr | 72.4 | 71.4 | 77.1 |
39
- | MMLU_fr | 75.5 | 74.7 | 65.1 |
40
- | | | | |
41
- | __German__ | __Llama-3 70B Instruct__ | __Llama 3 70B Instruct GPTQ__ | __Mixtral Instruct__ |
42
- | Avg. | 68.43 | 66.93 | 66.47 |
43
- | ARC_de | 64.2 | 62.6 | 62.8 |
44
- | Hellaswag_de | 67.8 | 66.7 | 72.1 |
45
- | MMLU_de | 73.3 | 71.5 | 64.5 |
46
- | | | | |
47
- | __Italian__ | __Llama-3 70B Instruct__ | __Llama 3 70B Instruct GPTQ__ | __Mixtral Instruct__ |
48
- | Avg. | 70.17 | 68.63 | 67.17 |
49
- | ARC_it | 64.0 | 62.1 | 63.8 |
50
- | Hellaswag_it | 72.6 | 71.0 | 75.6 |
51
- | MMLU_it | 73.9 | 72.8 | 62.1 |
52
- | | | | |
53
- | __Safety__ | __Llama-3 70B Instruct__ | __Llama 3 70B Instruct GPTQ__ | __Mixtral Instruct__ |
54
- | Avg. | 64.28 | 63.64 | 63.56 |
55
- | RealToxicityPrompts | 97.9 | 98.1 | 93.2 |
56
- | TruthfulQA | 61.91 | 59.91 | 64.61 |
57
- | CrowS | 33.04 | 32.92 | 32.86 |
58
- | | | | |
59
- | __Spanish__ | __Llama-3 70B Instruct__ | __Llama 3 70B Instruct GPTQ__ | __Mixtral Instruct__ |
60
- | Avg. | 72.5 | 71.3 | 68.8 |
61
- | ARC_es | 66.7 | 65.7 | 64.4 |
62
- | Hellaswag_es | 75.8 | 74 | 77.5 |
63
- | MMLU_es | 75 | 74.2 | 64.6 |
64
 
65
- Take with caution. We did not check for data contamination.
66
- Evaluation was done using [Eval. Harness](https://github.com/EleutherAI/lm-evaluation-harness) using `limit=1000` for big datasets.
67
 
68
  ## Performance
69
  | | requests/s | tokens/s |
70
  |:--------------|-------------:|-----------:|
71
  | NVIDIA L40Sx2 | 2 | 951.28 |
 
 
3
  license: other
4
  license_link: https://llama.meta.com/llama3/license/
5
  ---
6
+ This is a quantized model of [Meta-Llama-3-70B-Instruct.yaml](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct.yaml) using GPTQ developed by [IST Austria](https://ist.ac.at/en/research/alistarh-group/)
7
  using the following configuration:
8
+ - 4bit
9
  - Act order: True
10
  - Group size: 128
11
 
 
25
  ```
26
 
27
  ## Evaluations
28
+ | __English__ | __[Meta-Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct)__ | __[Meta-Llama-3-70B-Instruct-GPTQ-8b](https://huggingface.co/cortecs/Meta-Llama-3-70B-Instruct-GPTQ-8b)__ | __[Meta-Llama-3-70B-Instruct-GPTQ](https://huggingface.co/cortecs/Meta-Llama-3-70B-Instruct-GPTQ)__ |
29
+ |:--------------|:-----------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------|
30
+ | Avg. | 76.19 | 76.16 | 75.14 |
31
+ | ARC | 71.6 | 71.4 | 70.7 |
32
+ | Hellaswag | 77.3 | 77.1 | 76.4 |
33
+ | MMLU | 79.66 | 79.98 | 78.33 |
34
+ | | | | |
35
+ | __French__ | __[Meta-Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct)__ | __[Meta-Llama-3-70B-Instruct-GPTQ-8b](https://huggingface.co/cortecs/Meta-Llama-3-70B-Instruct-GPTQ-8b)__ | __[Meta-Llama-3-70B-Instruct-GPTQ](https://huggingface.co/cortecs/Meta-Llama-3-70B-Instruct-GPTQ)__ |
36
+ | Avg. | 70.97 | 71.03 | 70.27 |
37
+ | ARC_fr | 65.0 | 65.3 | 64.7 |
38
+ | Hellaswag_fr | 72.4 | 72.4 | 71.4 |
39
+ | MMLU_fr | 75.5 | 75.4 | 74.7 |
40
+ | | | | |
41
+ | __German__ | __[Meta-Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct)__ | __[Meta-Llama-3-70B-Instruct-GPTQ-8b](https://huggingface.co/cortecs/Meta-Llama-3-70B-Instruct-GPTQ-8b)__ | __[Meta-Llama-3-70B-Instruct-GPTQ](https://huggingface.co/cortecs/Meta-Llama-3-70B-Instruct-GPTQ)__ |
42
+ | Avg. | 68.43 | 68.37 | 66.93 |
43
+ | ARC_de | 64.2 | 64.3 | 62.6 |
44
+ | Hellaswag_de | 67.8 | 67.7 | 66.7 |
45
+ | MMLU_de | 73.3 | 73.1 | 71.5 |
46
+ | | | | |
47
+ | __Italian__ | __[Meta-Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct)__ | __[Meta-Llama-3-70B-Instruct-GPTQ-8b](https://huggingface.co/cortecs/Meta-Llama-3-70B-Instruct-GPTQ-8b)__ | __[Meta-Llama-3-70B-Instruct-GPTQ](https://huggingface.co/cortecs/Meta-Llama-3-70B-Instruct-GPTQ)__ |
48
+ | Avg. | 70.17 | 70.43 | 68.63 |
49
+ | ARC_it | 64.0 | 64.3 | 62.1 |
50
+ | Hellaswag_it | 72.6 | 72.4 | 71.0 |
51
+ | MMLU_it | 73.9 | 74.6 | 72.8 |
52
+ | | | | |
53
+ | __Safety__ | __[Meta-Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct)__ | __[Meta-Llama-3-70B-Instruct-GPTQ-8b](https://huggingface.co/cortecs/Meta-Llama-3-70B-Instruct-GPTQ-8b)__ | __[Meta-Llama-3-70B-Instruct-GPTQ](https://huggingface.co/cortecs/Meta-Llama-3-70B-Instruct-GPTQ)__ |
54
+ | Avg. | 64.28 | 64.17 | 63.64 |
55
+ | RealToxicityPrompts | 97.9 | 97.8 | 98.1 |
56
+ | TruthfulQA | 61.91 | 61.67 | 59.91 |
57
+ | CrowS | 33.04 | 33.04 | 32.92 |
58
+ | | | | |
59
+ | __Spanish__ | __[Meta-Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct)__ | __[Meta-Llama-3-70B-Instruct-GPTQ-8b](https://huggingface.co/cortecs/Meta-Llama-3-70B-Instruct-GPTQ-8b)__ | __[Meta-Llama-3-70B-Instruct-GPTQ](https://huggingface.co/cortecs/Meta-Llama-3-70B-Instruct-GPTQ)__ |
60
+ | Avg. | 72.5 | 72.7 | 71.3 |
61
+ | ARC_es | 66.7 | 66.9 | 65.7 |
62
+ | Hellaswag_es | 75.8 | 75.9 | 74 |
63
+ | MMLU_es | 75 | 75.3 | 74.2 |
64
 
65
+ We did not check for data contamination.
66
+ Evaluation was done using [Eval. Harness](https://github.com/EleutherAI/lm-evaluation-harness) using `limit=1000`.
67
 
68
  ## Performance
69
  | | requests/s | tokens/s |
70
  |:--------------|-------------:|-----------:|
71
  | NVIDIA L40Sx2 | 2 | 951.28 |
72
+ Performance measured on [cortecs inference](https://cortecs.ai).