|
--- |
|
datasets: |
|
- NeelNanda/pile-10k |
|
language: |
|
- en |
|
--- |
|
|
|
|
|
## Model Recipe Details |
|
|
|
This is an int4 model recipe with group_size 128 of [facebook/opt-1.3b](https://huggingface.co/facebook/opt-1.3b) generated by [intel/auto-round](https://github.com/intel/auto-round). |
|
Inference of this model is compatible with AutoGPTQ's Kernel. |
|
|
|
|
|
|
|
|
|
### Quantize the model |
|
|
|
Here is the sample command to reproduce the model |
|
|
|
```bash |
|
pip install auto-round |
|
auto-round |
|
--model facebook/opt-1.3b \ |
|
--device 0 \ |
|
--group_size 128 \ |
|
--bits 4 \ |
|
--iters 1000 \ |
|
--nsamples 512 \ |
|
--format 'auto_gptq' \ |
|
--minmax_lr 2e-3 \ |
|
--disable_quanted_input \ |
|
--output_dir "./tmp_autoround" \ |
|
``` |
|
|
|
## How to use |
|
|
|
### INT4 Inference with IPEX on Intel CPU |
|
Install the latest [Intel Extension for Pytorch](https://github.com/intel/intel-extension-for-pytorch) and [Intel Neural Compressor](https://github.com/intel/neural-compressor) |
|
|
|
```bash |
|
pip install torch --index-url https://download.pytorch.org/whl/cpu |
|
pip install intel_extension_for_pytorch |
|
pip install neural_compressor_pt |
|
``` |
|
|
|
```python |
|
from transformers import AutoTokenizer |
|
from neural_compressor.transformers import AutoModelForCausalLM |
|
|
|
## note: use quantized model directory name below |
|
model_name_or_path="./tmp_autoround/<model directory name>" |
|
q_model = AutoModelForCausalLM.from_pretrained(model_name_or_path) |
|
|
|
prompt = "Once upon a time, a little girl" |
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True) |
|
print(tokenizer.decode(q_model.generate(**tokenizer(prompt, return_tensors="pt").to(q_model.device),max_new_tokens=50)[0])) |
|
## Once upon a time, a little girl was born. She was a beautiful little girl, with a beautiful smile. She was a little girl who loved to play. She was a little girl who loved to sing.She was a little girl who loved to dance. |
|
``` |
|
|
|
### INT4 Inference on Intel Gaudi Accelerator |
|
docker image with Gaudi Software Stack is recommended. More details can be found in [Gaudi Guide](https://docs.habana.ai/en/latest/). |
|
|
|
```python |
|
import habana_frameworks.torch.core as htcore |
|
from neural_compressor.torch.quantization import load |
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
|
|
## note: use quantized model directory name below |
|
model_name_or_path="./tmp_autoround/<model directory name>" |
|
|
|
model = load( |
|
model_name_or_path=model_name_or_path, |
|
format="huggingface", |
|
device="hpu" |
|
) |
|
|
|
prompt = "Once upon a time, a little girl" |
|
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path) |
|
print(tokenizer.decode(model.generate(**tokenizer(prompt, return_tensors="pt").to("hpu"),max_new_tokens=50)[0])) |
|
|
|
``` |
|
|
|
## Accuracy Result |
|
|
|
|
|
| Metric <img width=200> | FP16 <img width=200> | INT4 <img width=200> | |
|
| :------------------ | :---------- | :---------- | |
|
| Avg. | 0.4405 | 0.4315 | |
|
| mmlu | 0.2502 | 0.2510 | |
|
| lambada_openai | 0.5789 | 0.5389 | |
|
| hellaswag | 0.4153 | 0.4076 | |
|
| winogrande | 0.5927 | 0.5848 | |
|
| piqa | 0.7165 | 0.7067 | |
|
| truthfulqa_mc1 | 0.2362 | 0.2338 | |
|
| openbookqa | 0.2320 | 0.2140 | |
|
| boolq | 0.5765 | 0.5786 | |
|
| arc_easy | 0.5724 | 0.5673 | |
|
| arc_challenge | 0.2346 | 0.2321 | |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Caveats and Recommendations |
|
|
|
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. |
|
|
|
Here are a couple of useful links to learn more about Intel's AI software: |
|
|
|
* Intel Neural Compressor [link](https://github.com/intel/neural-compressor) |
|
* Intel Extension for Transformers [link](https://github.com/intel/intel-extension-for-transformers) |
|
|
|
|
|
|
|
## Disclaimer |
|
|
|
The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please consult an attorney before using this model for commercial purposes. |
|
|
|
|
|
|
|
## Cite |
|
|
|
@article{cheng2023optimize, title={Optimize weight rounding via signed gradient descent for the quantization of llms}, author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao}, journal={arXiv preprint arXiv:2309.05516}, year={2023} } |
|
|
|
[arxiv](https://arxiv.org/abs/2309.05516) [github](https://github.com/intel/auto-round) |
