This repo contains a low-rank adapter for LLaMA-7b fit on the Stanford Alpaca dataset.
This version of the weights was trained with the following hyperparameters:
[Cleaned dataset](https://github.com/gururise/AlpacaDataCleaned): Snapshot March 31, 2023
Epochs: 3
Validation set size: 2000
Batch size: 128
Micro batch size: 12
Cutoff length: 512
Learning rate: 3e-4
Lora r: 8
Lora target modules: q_proj, v_proj
That is:
python finetune.py
--base_model='decapoda-research/llama-7b-hf'
--data_path 'yahma/alpaca-cleaned'
--num_epochs=3
--cutoff_len=512
--output_dir='./lora-alpaca'
--lora_target_modules='[q_proj,v_proj]'
--lora_r=8
--val_set_size 2000
--micro_batch_size=12