File size: 372 Bytes
4550fe1 604068d |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
---
license: mit
language:
- en
- zh
base_model:
- deepseek-ai/DeepSeek-V3
pipeline_tag: text-generation
library_name: transformers
---
# DeepSeek V3 AWQ
AWQ of the DeepSeek V3 chat model.
This quant modified some of the model code to fix the overflow issue when using float16.
Tested on vLLM with 8x H100, inference speed 5 tokens/s with batch size 1 and short prompts. |