abhinavkulkarni
/

tiiuae-falcon-40b-instruct-w4-g128-awq

Text Generation

text-generation-inference

Model card Files Files and versions Community

mike-ravkine commited on Jul 13, 2023

Commit

3cb2c18

•

1 Parent(s): 4201307

Fix q_group_size

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -54,7 +54,7 @@ tokenizer = AutoTokenizer.from_pretrained(config.tokenizer_name)
 w_bit = 4
 q_config = {
     "zero_point": True,
-    "q_group_size": 64,
 }
 load_quant = snapshot_download('abhinavkulkarni/tiiuae-falcon-40b-instruct-w4-g128-awq')

 w_bit = 4
 q_config = {
     "zero_point": True,
+    "q_group_size": 128,
 }
 load_quant = snapshot_download('abhinavkulkarni/tiiuae-falcon-40b-instruct-w4-g128-awq')