fix: `train_on_inputs: true` ignored for sharegpt (#1045) [skip ci]

fix double eos token for chatml (#1054) [skip ci]

Create preprocess CLI (#785)

improve: Enhance code readability of prompt_tokenizers.py (#707)

misc sharegpt fixes (#723)

don't strip the prompt for check since we don't strip to tokenize anymore (#650)

use fastchat conversations template (#578)

better handling and logging of empty sharegpt turns (#603)

split completion text to sequence_len (#616)

improve handling for empty text on the tokenization step (#502)

support custom field for completion from yml (#580)

improve llama pad token handling (#475)

gracefully handle empty input (#442)

better handling of empty input ids when tokenizing (#395)

better handling since xgen tokenizer breaks with convert_tokens_to_ids

Adding logging enhancement

Fix typing list

initial wip to get sys prompt from dataset

bugfix for potential off by one

Fix sharegpt prompt

Fix security issue or ignore false positives

Apply isort then black

Fix mypy typing

Fix unsupported operand type(s) for |

Refactor duplicate code between Prompter and Pygmalion

Lint prompt_tokenizers

refactor conversation plucking in sharegpt

apply black formatting

tokenization fixes

optionally be able to specify alpaca or chat style prompts

concise multiple choice and tldr summarize

add alpaca multiple choice instruct dataset support

fix prompters, especially the sharegpt prompter

black formatting

Rename variable to use same convention

Add CompletionPrompt type

Jeopardy bot! (#17)

WIP large refactor to make finetune script a little more manageable (#3)

add support for alpaca reflect training (#2)

Tokenization open assistant (#1)

suppport for alpaca-like instruction datasets without inputs

config chooser, update readme instructions, device config, llama flash attention, debug out the labels, fix config key checks, other bugfixes

black formatting

make it work with pythia in the cloud

WIP for axolotl trainer