OPEA
/

Safetensors
cicdatopea commited on
Commit
b7b236d
1 Parent(s): aa98c01

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +3 -3
README.md CHANGED
@@ -16,7 +16,7 @@ CPU/ CUDA requires auto-round version>0.3.1
16
  ```python
17
  from auto_round import AutoRoundConfig ##must import for auto-round format
18
  from transformers import AutoModelForCausalLM,AutoTokenizer
19
- quantized_model_dir = "OPEA/Qwen2.5-32B-Instruct-int4-inc"
20
  tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir)
21
 
22
  model = AutoModelForCausalLM.from_pretrained(
@@ -127,7 +127,7 @@ prompt = "请简短介绍一下阿里巴巴公司"
127
  pip3 install lm-eval==0.4.5
128
 
129
  ```bash
130
- auto-round --model "OPEA/Qwen2.5-32B-Instruct-int4-inc" --eval --eval_bs 16 --tasks leaderboard_ifeval,leaderboard_mmlu_pro,gsm8k,lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,arc_easy,arc_challenge,cmmlu,ceval-valid
131
  ```
132
 
133
  | Metric | BF16 | INT4 |
@@ -156,7 +156,7 @@ auto-round --model "OPEA/Qwen2.5-32B-Instruct-int4-inc" --eval --eval_bs 16 --t
156
 
157
  Here is the sample command to generate the model.
158
 
159
- For symmetric quantization, we found overflow/NAN will occur for some backends, so better fallback some layers. auto_round requires version >0.4.1
160
 
161
  ```bash
162
  auto-round \
 
16
  ```python
17
  from auto_round import AutoRoundConfig ##must import for auto-round format
18
  from transformers import AutoModelForCausalLM,AutoTokenizer
19
+ quantized_model_dir = "OPEA/Qwen2.5-32B-Instruct-int4-sym-mixed-inc"
20
  tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir)
21
 
22
  model = AutoModelForCausalLM.from_pretrained(
 
127
  pip3 install lm-eval==0.4.5
128
 
129
  ```bash
130
+ auto-round --model "OPEA/Qwen2.5-32B-Instruct-int4-sym-mixed-inc" --eval --eval_bs 16 --tasks leaderboard_ifeval,leaderboard_mmlu_pro,gsm8k,lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,arc_easy,arc_challenge,cmmlu,ceval-valid
131
  ```
132
 
133
  | Metric | BF16 | INT4 |
 
156
 
157
  Here is the sample command to generate the model.
158
 
159
+ For symmetric quantization, we found overflow/NAN will occur for some backends, so better fallback some layers. auto_round requires version >=0.4.1
160
 
161
  ```bash
162
  auto-round \