Commit
•
df3a616
1
Parent(s):
9d986f6
Update README.md
Browse files
README.md
CHANGED
|
@@ -74,19 +74,22 @@ pip3 install lm-eval==0.4.5
|
|
| 74 |
auto-round --model "OPEA/Falcon3-3B-Base-int4-sym-inc" --eval --eval_bs 16 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,arc_easy,arc_challenge,mmlu
|
| 75 |
```
|
| 76 |
|
| 77 |
-
| Metric |
|
| 78 |
-
| :-------------------------- |
|
| 79 |
-
| Avg | 0.
|
| 80 |
-
|
|
| 81 |
-
|
|
| 82 |
-
|
|
| 83 |
-
|
|
| 84 |
-
|
|
| 85 |
-
|
|
| 86 |
-
|
|
| 87 |
-
|
|
| 88 |
-
|
|
| 89 |
-
|
|
|
|
|
|
|
|
|
|
|
| 90 |
|
| 91 |
### Generate the model
|
| 92 |
|
|
|
|
| 74 |
auto-round --model "OPEA/Falcon3-3B-Base-int4-sym-inc" --eval --eval_bs 16 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,arc_easy,arc_challenge,mmlu
|
| 75 |
```
|
| 76 |
|
| 77 |
+
| Metric | BF16 | INT4 |
|
| 78 |
+
| :-------------------------- | :-----------------------: | :----------------------: |
|
| 79 |
+
| Avg | 0.5203 | 0.5142 |
|
| 80 |
+
| leaderboard_mmlu_pro | 0.2204 | 0.2152 |
|
| 81 |
+
| leaderboard_ifeval | 0.2893=(0.3549+0.2237)/2 | 0.2917=(0.3597+0.2237)/2 |
|
| 82 |
+
| gsm8k(5shot) strict match | 0.4375 | 0.4261 |
|
| 83 |
+
| mmlu | 0.5192 | 0.5127 |
|
| 84 |
+
| lambada_opeai | 0.6109 | 0.6103 |
|
| 85 |
+
| hellaswage | 0.5357 | 0.5285 |
|
| 86 |
+
| winogrande | 0.6614 | 0.6638 |
|
| 87 |
+
| piqa | 0.7753 | 0.7715 |
|
| 88 |
+
| truthfulqa_mc1 | 0.2803 | 0.2742 |
|
| 89 |
+
| openbookqa | 0.3560 | 0.3360 |
|
| 90 |
+
| boolq | 0.8040 | 0.7884 |
|
| 91 |
+
| arc_easy | 0.7896 | 0.7845 |
|
| 92 |
+
| arc_challenge | 0.4838 | 0.4812 |
|
| 93 |
|
| 94 |
### Generate the model
|
| 95 |
|