lvkaokao commited on
Commit
77439b2
1 Parent(s): 46dba76

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +17 -17
README.md CHANGED
@@ -65,24 +65,24 @@ print(tokenizer.decode(model.generate(**tokenizer("There is a girl who likes adv
65
 
66
  Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, we used the git id f3b7917091afba325af3980a35d8a6dcba03dc3f
67
 
68
- ~~bash
69
  lm_eval --model hf --model_args pretrained="Intel/neural-chat-v3-1-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge --batch_size 128
70
- ~~
71
-
72
- | Metric | FP16 | INT4 | qdq |
73
- | -------------- | ------ | ------ | ------ |
74
- | Avg. | 0.6769 | 0.6721 | 0.6718 |
75
- | mmlu | 0.5919 | 0.5862 | 0.5863 |
76
- | lambada_openai | 0.7394 | 0.7337 | 0.7334 |
77
- | hellaswag | 0.6323 | 0.6272 | 0.6267 |
78
- | winogrande | 0.7687 | 0.7577 | 0.7577 |
79
- | piqa | 0.8161 | 0.8150 | 0.8150 |
80
- | truthfulqa_mc1 | 0.4431 | 0.4394 | 0.4382 |
81
- | openbookqa | 0.3760 | 0.3700 | 0.3680 |
82
- | boolq | 0.8783 | 0.8743 | 0.8743 |
83
- | rte | 0.7690 | 0.7726 | 0.7726 |
84
- | arc_easy | 0.8413 | 0.8384 | 0.8388 |
85
- | arc_challenge | 0.5896 | 0.5785 | 0.5785 |
86
 
87
 
88
 
 
65
 
66
  Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, we used the git id f3b7917091afba325af3980a35d8a6dcba03dc3f
67
 
68
+ ```bash
69
  lm_eval --model hf --model_args pretrained="Intel/neural-chat-v3-1-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge --batch_size 128
70
+ ```
71
+
72
+ | Metric | FP16 | INT4 |
73
+ | -------------- | ------ | ------ |
74
+ | Avg. | 0.6769 | 0.6721 |
75
+ | mmlu | 0.5919 | 0.5862 |
76
+ | lambada_openai | 0.7394 | 0.7337 |
77
+ | hellaswag | 0.6323 | 0.6272 |
78
+ | winogrande | 0.7687 | 0.7577 |
79
+ | piqa | 0.8161 | 0.8150 |
80
+ | truthfulqa_mc1 | 0.4431 | 0.4394 |
81
+ | openbookqa | 0.3760 | 0.3700 |
82
+ | boolq | 0.8783 | 0.8743 |
83
+ | rte | 0.7690 | 0.7726 |
84
+ | arc_easy | 0.8413 | 0.8384 |
85
+ | arc_challenge | 0.5896 | 0.5785 |
86
 
87
 
88