Update README.md
Browse files
README.md
CHANGED
@@ -65,24 +65,24 @@ print(tokenizer.decode(model.generate(**tokenizer("There is a girl who likes adv
|
|
65 |
|
66 |
Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, we used the git id f3b7917091afba325af3980a35d8a6dcba03dc3f
|
67 |
|
68 |
-
|
69 |
lm_eval --model hf --model_args pretrained="Intel/neural-chat-v3-1-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge --batch_size 128
|
70 |
-
|
71 |
-
|
72 |
-
| Metric | FP16 | INT4 |
|
73 |
-
| -------------- | ------ | ------ |
|
74 |
-
| Avg. | 0.6769 | 0.6721 |
|
75 |
-
| mmlu | 0.5919 | 0.5862 |
|
76 |
-
| lambada_openai | 0.7394 | 0.7337 |
|
77 |
-
| hellaswag | 0.6323 | 0.6272 |
|
78 |
-
| winogrande | 0.7687 | 0.7577 |
|
79 |
-
| piqa | 0.8161 | 0.8150 |
|
80 |
-
| truthfulqa_mc1 | 0.4431 | 0.4394 |
|
81 |
-
| openbookqa | 0.3760 | 0.3700 |
|
82 |
-
| boolq | 0.8783 | 0.8743 |
|
83 |
-
| rte | 0.7690 | 0.7726 |
|
84 |
-
| arc_easy | 0.8413 | 0.8384 |
|
85 |
-
| arc_challenge | 0.5896 | 0.5785 |
|
86 |
|
87 |
|
88 |
|
|
|
65 |
|
66 |
Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, we used the git id f3b7917091afba325af3980a35d8a6dcba03dc3f
|
67 |
|
68 |
+
```bash
|
69 |
lm_eval --model hf --model_args pretrained="Intel/neural-chat-v3-1-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge --batch_size 128
|
70 |
+
```
|
71 |
+
|
72 |
+
| Metric | FP16 | INT4 |
|
73 |
+
| -------------- | ------ | ------ |
|
74 |
+
| Avg. | 0.6769 | 0.6721 |
|
75 |
+
| mmlu | 0.5919 | 0.5862 |
|
76 |
+
| lambada_openai | 0.7394 | 0.7337 |
|
77 |
+
| hellaswag | 0.6323 | 0.6272 |
|
78 |
+
| winogrande | 0.7687 | 0.7577 |
|
79 |
+
| piqa | 0.8161 | 0.8150 |
|
80 |
+
| truthfulqa_mc1 | 0.4431 | 0.4394 |
|
81 |
+
| openbookqa | 0.3760 | 0.3700 |
|
82 |
+
| boolq | 0.8783 | 0.8743 |
|
83 |
+
| rte | 0.7690 | 0.7726 |
|
84 |
+
| arc_easy | 0.8413 | 0.8384 |
|
85 |
+
| arc_challenge | 0.5896 | 0.5785 |
|
86 |
|
87 |
|
88 |
|