license: other
language:
- en
pipeline_tag: text2text-generation
tags:
- alpaca
- llama
- chat
- gpt4
inference: false
This is a 4bit 128g GPTQ of chansung's gpt4-alpaca-lora-13b.
More details will be put in this README tomorrow. Until then, please see one of my other GPTQ repos for more instructions.
Command to create was:
cd gptq-safe && CUDA_VISIBLE_DEVICES=0 python3 llama.py /content/gpt4-alpaca-lora-13B-HF c4 --wbits 4 --true-sequential --act-order --groupsize 128 --save_safetensors /content/gpt4-alpaca-lora-13B-GPTQ-4bit-128g.safetensors
Note that as --act-order was used, this will not work with ooba's fork of GPTQ. You must use the qwopqwop repo as of April 13th.
Command to clone the correct GPTQ-for-LLaMa repo for inference using llama_inference.py, or in text-generation-webui:
git clone -n https://github.com/qwopqwop200/GPTQ-for-LLaMa gptq-safe
cd gptq-safe
git checkout 58c8ab4c7aaccc50f507fd08cce941976affe5e0
There is also a no-act-order.safetensors file which will work with oobabooga's fork of GPTQ-for-LLaMa; it does not require the latest GPTQ code.
Original model card is below
This repository comes with LoRA checkpoint to make LLaMA into a chatbot like language model. The checkpoint is the output of instruction following fine-tuning process with the following settings on 8xA100(40G) DGX system.
- Training script: borrowed from the official Alpaca-LoRA implementation
- Training script:
python finetune.py \
--base_model='decapoda-research/llama-30b-hf' \
--data_path='alpaca_data_gpt4.json' \
--num_epochs=10 \
--cutoff_len=512 \
--group_by_length \
--output_dir='./gpt4-alpaca-lora-30b' \
--lora_target_modules='[q_proj,k_proj,v_proj,o_proj]' \
--lora_r=16 \
--batch_size=... \
--micro_batch_size=...
You can find how the training went from W&B report here.