csdc-atl
/

Baichuan2-13B-Chat-GPTQ-Int4

Text Generation

text-generation-inference

Inference Endpoints

4-bit precision

Model card Files Files and versions Community

xiaotinghe commited on Sep 20, 2023

Commit

d626856

•

1 Parent(s): 6004920

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -33,7 +33,7 @@ tasks:
 |---|---|---|---|---|---|
 | [Baichuan2-13B-Chat](https://huggingface.co/baichuan-inc/Baichuan2-13B-Chat) | 40.25 | 56.33 | 58.44 | 27.79g | 31.55 tokens/s |
 | [Baichuan2-13B-Chat-4bits](https://huggingface.co/baichuan-inc/Baichuan2-13B-Chat-4bits) | ~ | ~ | ~ | 9.08g | 18.45 tokens/s |
-| [GPTQ-4bit-32g](https://huggingface.co/csdc-atl/Baichuan2-13B-Chat-GPTQ-Int4/tree/4bit-32g) | ~ | ~ | ~ | 9.87g | 27.35(hf) \ 38.28(autogptq) tokens/s   |
 | [GPTQ-4bit-128g](https://huggingface.co/csdc-atl/Baichuan2-13B-Chat-GPTQ-Int4/tree/main) | 38.78 | 56.42 | 57.78 | 9.14g | 28.74(hf) \ 39.24(autogptq) tokens/s  |
 <!-- README_GPTQ.md-provided-files end -->

 |---|---|---|---|---|---|
 | [Baichuan2-13B-Chat](https://huggingface.co/baichuan-inc/Baichuan2-13B-Chat) | 40.25 | 56.33 | 58.44 | 27.79g | 31.55 tokens/s |
 | [Baichuan2-13B-Chat-4bits](https://huggingface.co/baichuan-inc/Baichuan2-13B-Chat-4bits) | ~ | ~ | ~ | 9.08g | 18.45 tokens/s |
+| [GPTQ-4bit-32g](https://huggingface.co/csdc-atl/Baichuan2-13B-Chat-GPTQ-Int4/tree/4bit-32g) | 38.64 | 57.18 | 57.47 | 9.87g | 27.35(hf) \ 38.28(autogptq) tokens/s   |
 | [GPTQ-4bit-128g](https://huggingface.co/csdc-atl/Baichuan2-13B-Chat-GPTQ-Int4/tree/main) | 38.78 | 56.42 | 57.78 | 9.14g | 28.74(hf) \ 39.24(autogptq) tokens/s  |
 <!-- README_GPTQ.md-provided-files end -->