cicdatopea
commited on
Commit
•
b7b236d
1
Parent(s):
aa98c01
Update README.md
Browse files
README.md
CHANGED
@@ -16,7 +16,7 @@ CPU/ CUDA requires auto-round version>0.3.1
|
|
16 |
```python
|
17 |
from auto_round import AutoRoundConfig ##must import for auto-round format
|
18 |
from transformers import AutoModelForCausalLM,AutoTokenizer
|
19 |
-
quantized_model_dir = "OPEA/Qwen2.5-32B-Instruct-int4-inc"
|
20 |
tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir)
|
21 |
|
22 |
model = AutoModelForCausalLM.from_pretrained(
|
@@ -127,7 +127,7 @@ prompt = "请简短介绍一下阿里巴巴公司"
|
|
127 |
pip3 install lm-eval==0.4.5
|
128 |
|
129 |
```bash
|
130 |
-
auto-round --model "OPEA/Qwen2.5-32B-Instruct-int4-inc" --eval --eval_bs 16 --tasks leaderboard_ifeval,leaderboard_mmlu_pro,gsm8k,lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,arc_easy,arc_challenge,cmmlu,ceval-valid
|
131 |
```
|
132 |
|
133 |
| Metric | BF16 | INT4 |
|
@@ -156,7 +156,7 @@ auto-round --model "OPEA/Qwen2.5-32B-Instruct-int4-inc" --eval --eval_bs 16 --t
|
|
156 |
|
157 |
Here is the sample command to generate the model.
|
158 |
|
159 |
-
For symmetric quantization, we found overflow/NAN will occur for some backends, so better fallback some layers. auto_round requires version
|
160 |
|
161 |
```bash
|
162 |
auto-round \
|
|
|
16 |
```python
|
17 |
from auto_round import AutoRoundConfig ##must import for auto-round format
|
18 |
from transformers import AutoModelForCausalLM,AutoTokenizer
|
19 |
+
quantized_model_dir = "OPEA/Qwen2.5-32B-Instruct-int4-sym-mixed-inc"
|
20 |
tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir)
|
21 |
|
22 |
model = AutoModelForCausalLM.from_pretrained(
|
|
|
127 |
pip3 install lm-eval==0.4.5
|
128 |
|
129 |
```bash
|
130 |
+
auto-round --model "OPEA/Qwen2.5-32B-Instruct-int4-sym-mixed-inc" --eval --eval_bs 16 --tasks leaderboard_ifeval,leaderboard_mmlu_pro,gsm8k,lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,arc_easy,arc_challenge,cmmlu,ceval-valid
|
131 |
```
|
132 |
|
133 |
| Metric | BF16 | INT4 |
|
|
|
156 |
|
157 |
Here is the sample command to generate the model.
|
158 |
|
159 |
+
For symmetric quantization, we found overflow/NAN will occur for some backends, so better fallback some layers. auto_round requires version >=0.4.1
|
160 |
|
161 |
```bash
|
162 |
auto-round \
|