| This is 4-bit quantization of [chavinlo/alpaca-native](https://huggingface.co/chavinlo/alpaca-native) (`cecc16d`) via [qwopqwop200/GPTQ-for-LLaMa](https://github.com/qwopqwop200/GPTQ-for-LLaMa) (`5cdfad2`). | |
| Quantization invoked as such: | |
| `llama.py /output/path c4 --wbits 4 --groupsize 128 --save alpaca7b-4bit.pt` | |
| Inference example from the GPTQ repo and commit referenced above: | |
| ``` | |
| (gptq) [root@gpu03 GPTQ-for-LLaMa]# CUDA_VISIBLE_DEVICES=0 python llama_inference.py /root/alpaca-native-4bit --wbits 4 --groupsize 128 --load /root/alpaca-native-4bit/alpaca7b-4bit.pt --max_length 300 --text "$(cat test_prompt.txt)" | |
| Loading model ... | |
| Done. | |
| ### Instruction: What is an alpaca? How is it different from a llama? | |
| ### Response: Alpacas are soft and gentle, while llamas are stubborn and independent.</s> | |
| (gptq) [root@gpu03 GPTQ-for-LLaMa]# CUDA_VISIBLE_DEVICES=0 python llama_inference.py /root/alpaca-native-4bit --wbits 4 --groupsize 128 --load /root/alpaca-native-4bit/alpaca7b-4bit.pt --max_length 300 --text "$(cat test_prompt.txt)" | |
| Loading model ... | |
| Done. | |
| ### Instruction: What is an alpaca? How is it different from a llama? | |
| ### Response: An alpaca is a small, domesticated species of livestock from the Andes region of South America. It is typically kept as a pet, and its fibers can be used for various purposes, such as making clothing and crafts. Alpacas are typically brown or black, and their ears and tails are often moved. | |
| Although it is different from a llama, the two animals are often compared to when referring to their behavior.</s> | |
| (gptq) [root@gpu03 GPTQ-for-LLaMa]# md5sum /root/alpaca-native-4bit/alpaca7b-4bit.pt | |
| 74849953cc54e313b972d2cc9a05c24b /root/alpaca-native-4bit/alpaca7b-4bit.pt | |
| (gptq) [root@gpu03 GPTQ-for-LLaMa]# | |
| ``` |