README.md

---
license: mit
datasets:
- yahma/alpaca-cleaned
---
This repo contains a low-rank adapter for LLaMA-7b fit on the Cleaned Alpaca dataset (with the new GPT-4 training data).

This version of the weights was trained with the following hyperparameters:

    Cleaned dataset: Snapshot April 8, 2023
    Epochs: 6 (Checkpoint with lowest eval loss at 3.6 epochs uploaded here)
    Validation set size: 1500
    Batch size: 128
    Micro batch size: 8
    Cutoff length: 512
    Learning rate: 3e-4
    Lora r: 16
    Lora target modules: q_proj, k_proj, v_proj, o_proj

That is:

python finetune.py \
    --base_model='yahma/llama-7b-hf' \
    --data_path 'yahma/alpaca-cleaned' \
    --num_epochs=6 \
    --cutoff_len=512 \
    --output_dir='./lora-alpaca' \
    --lora_target_modules='[q_proj,k_proj, v_proj, o_proj]' \
    --lora_r=16 \
    --val_set_size 1500 \
    --micro_batch_size=8