attempt xformers hijack attention

WIP large refactor to make finetune script a little more manageable (#3)

add support for alpaca reflect training (#2)

update readme

Tokenization open assistant (#1)

fix llama check

update readme

fix conditional check to prevent always using 4bit

imrpove llama check and fix safetensors file check

suppport for alpaca-like instruction datasets without inputs

casts the prepared data to int16 (doesn't help with training memory)

bugfixes

fix lora target module, require explicit flash attention, fix min logging steps, don't use adam8bit for int4, hash prepared datasets, support hf hub datasets

fix install to work with latest alpaca lora 4bit

4bit quantized support (wip)

cleanup, prep for 4bit quant support

deepspeed doesn't work with flash-attn, and the gpu savings w flash attn are better than the deepspeed headaches

fix logging

prepare datasets only flag

add llama 7b config and fiz lora_fan_in_fan_out for llama (copy pasta bug)

configure log level, add llama 7b config

more logging, wandb fixes

refactor trainer setup to account for deepspeed integration

improve prepared dataset loading, fix inference

helpful info output

fix issue with completed model being empty

various bugfixes

bettter handling of llama model import

more fixes and prep for llama training

config chooser, update readme instructions, device config, llama flash attention, debug out the labels, fix config key checks, other bugfixes

black formatting

make it work with pythia in the cloud

WIP for axolotl trainer

initial commit of README