File size: 1,549 Bytes
1a570cf cac9082 1a570cf cac9082 1a570cf 76c5e49 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 |
---
license: apache-2.0
language:
- en
pipeline_tag: text-generation
tags:
- AutoGPTQ
- 4bit
- GPTQ
---
Model created using [AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ) on a [GPT-2](https://huggingface.co/gpt2) model with 4-bit quantization.
You can load this model with the AutoGPTQ library, installed with the following command:
```
pip install auto-gptq
```
You can then download the model from the hub using the following code:
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
model_name = "mlabonne/gpt2-GPTQ-4bit"
tokenizer = AutoTokenizer.from_pretrained(model_name)
quantize_config = BaseQuantizeConfig.from_pretrained(model_name)
model = AutoGPTQForCausalLM.from_quantized(model_name,
model_basename="gptq_model-4bit-128g",
device="cuda:0",
use_triton=True,
use_safetensors=True,
quantize_config=quantize_config)
```
This model works with the traditional [Text Generation pipeline](https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.TextGenerationPipeline).
Example of generation with the input text "I have a dream":
```
I have a dream. I want someone with my face, and what I have. I want to go home. I want to be alive. I want to see my children. I dream if I have the spirit, my body, my voice,
``` |