Merge pull request #13 from winglian/dev

Merge pull request #12 from NanoCode012/feat/eval_config

Add eval_batch_size for evaluation

fix log sweep lr

support for multi line inference input, log sweep over learning rates

fix adam bnb optimizer grouped parameters, fix peft model 8bit conversion logic, black formatting

install peft from main branch

support llama-adapter zero init attention

use prebuilt wheels for flash-attn and deepspeed

fdsp config dict fix, todo list, add torchdistx support

8bit and deepspeed changes

update ds_config

don't load models in 8bit unless they are using an adapter, also fix tokenizer load in exceptional case

fix fsdp training args

fix for zero value warmup steps

fix sharegpt tokenization, refactor tokenization debugging

wire up gradient checkpointing for 4bit

Merge pull request #9 from winglian/dev

fix dataset handling, support galactica

tweaks to data loading, 8 bit adam, accelerate and deepspeed

shuffle and split dataset after save/load

fix sharegpt handling from hf, don't worry about loading llama if using earlier transformers release

stablelm support

various bugfixes

ignore config, add python 3.9 (#8)

fix bug when model_type not explicitly passed

improve inference

fix runpod script

cleanup empty lines, tweak env for runpod setup

handle empty lines

quickstart instructions for starting from runpod (#5)

update readme w compat matrix

attempt xformers hijack attention

WIP large refactor to make finetune script a little more manageable (#3)

add support for alpaca reflect training (#2)

update readme

Tokenization open assistant (#1)

fix llama check

update readme

fix conditional check to prevent always using 4bit

imrpove llama check and fix safetensors file check

suppport for alpaca-like instruction datasets without inputs

casts the prepared data to int16 (doesn't help with training memory)

bugfixes

fix lora target module, require explicit flash attention, fix min logging steps, don't use adam8bit for int4, hash prepared datasets, support hf hub datasets

fix install to work with latest alpaca lora 4bit

4bit quantized support (wip)

cleanup, prep for 4bit quant support

deepspeed doesn't work with flash-attn, and the gpu savings w flash attn are better than the deepspeed headaches

fix logging