|
--- |
|
license: apache-2.0 |
|
language: |
|
- en |
|
pipeline_tag: text2text-generation |
|
tags: |
|
- alpaca |
|
- llama |
|
- chat |
|
- gpt4 |
|
--- |
|
|
|
This repository comes with LoRA checkpoint to make LLaMA into a chatbot like language model. The checkpoint is the output of instruction following fine-tuning process with the following settings on 8xA100(40G) DGX system. |
|
- Training script: borrowed from the official [Alpaca-LoRA](https://github.com/tloen/alpaca-lora) implementation |
|
- Training script: |
|
```shell |
|
python finetune.py \ |
|
--base_model='decapoda-research/llama-30b-hf' \ |
|
--data_path='alpaca_data_gpt4.json' \ |
|
--num_epochs=10 \ |
|
--cutoff_len=512 \ |
|
--group_by_length \ |
|
--output_dir='./gpt4-alpaca-lora-30b' \ |
|
--lora_target_modules='[q_proj,k_proj,v_proj,o_proj]' \ |
|
--lora_r=16 \ |
|
--batch_size=... \ |
|
--micro_batch_size=... |
|
``` |
|
|
|
You can find how the training went from W&B report [here](https://wandb.ai/chansung18/gpt4_alpaca_lora/runs/w3syd157?workspace=user-chansung18). |
|
|
