Merge pull request #13 from winglian/dev

Add eval_batch_size for evaluation

fix log sweep lr

support for multi line inference input, log sweep over learning rates

fix adam bnb optimizer grouped parameters, fix peft model 8bit conversion logic, black formatting

support llama-adapter zero init attention

fdsp config dict fix, todo list, add torchdistx support

8bit and deepspeed changes

don't load models in 8bit unless they are using an adapter, also fix tokenizer load in exceptional case

fix fsdp training args

fix for zero value warmup steps

fix sharegpt tokenization, refactor tokenization debugging

wire up gradient checkpointing for 4bit

fix dataset handling, support galactica

tweaks to data loading, 8 bit adam, accelerate and deepspeed

shuffle and split dataset after save/load

fix sharegpt handling from hf, don't worry about loading llama if using earlier transformers release

various bugfixes

fix bug when model_type not explicitly passed

improve inference

quickstart instructions for starting from runpod (#5)

attempt xformers hijack attention

WIP large refactor to make finetune script a little more manageable (#3)

add support for alpaca reflect training (#2)

Tokenization open assistant (#1)

suppport for alpaca-like instruction datasets without inputs

casts the prepared data to int16 (doesn't help with training memory)

4bit quantized support (wip)

various bugfixes

config chooser, update readme instructions, device config, llama flash attention, debug out the labels, fix config key checks, other bugfixes

black formatting

make it work with pythia in the cloud

WIP for axolotl trainer