metadata

license: other
inference: false

OpenAssistant LLaMA 30B SFT 7 GPTQ

This in a repo of GPTQ format 4bit quantised models for OpenAssistant's LLaMA 30B SFT 7.

It is the result of merging the XORs from the above repo with the original Llama 30B weights, and then quantising to 4bit GPU inference using GPTQ-for-LLaMa.

This is epoch 7 of OpenAssistant's training of their Llama 30B model.

Please note that these models will need 24GB VRAM or greater to use effectively

Repositories available

PROMPT TEMPLATE

This model requires the following prompt template:

<|prompter|> prompt goes here
<|assistant|>:

CHOICE OF MODELS

Three sets of models are provided:

Groupsize = None
- Should work reliably in 24GB VRAM
- Uses --act-order for the best possible inference quality given its lack of group_size.
Groupsize = 1024
- Theoretically higher inference accuracy
- May OOM on long context lengths in 24GB VRAM
Groupsize = 128
- Optimal setting for highest inference quality
- Will definitely need more than 24GB VRAM on longer context lengths (1000-1500+ tokens returned)

For the 128g and 1024g models, two versions are available:

compat.no-act-order.safetensor
- Works with all versions of GPTQ-for-LLaMa, including the version in text-generation-webui one-click-installers
latest.act-order.safetensors
- uses --act-order for higher inference quality
- requires more recent GPTQ-for-LLaMa code, therefore will not currently work with one-click-installers

HOW TO CHOOSE YOUR MODEL

I have used branches to separate the models. This means you can clone the branch you want and not got model files you don't need.

If you have 24GB VRAM you are strongly recommended to use the file in main, with group_size = None. This is fully compatible, and won't OOM.

Branch: main = groupsize None, OpenAssistant-SFT-7-Llama-30B-GPTQ-4bit.safetensors file
Branch: 1024-compat = groupsize 1024, compat.no-act-order.safetensors file
Branch: 1024-latest = groupsize 1024, latest.act-order.safetensors file
Branch: 128-compat = groupsize 128, compat.no-act-order.safetensors file
Branch: 128-latest = groupsize 128, latest.act-order.safetensors file

How to easily download and run the 1024g compat model in text-generation-webui

Open the text-generation-webui UI as normal.

Click the Model tab.
Under Download custom model or LoRA, enter TheBloke/OpenAssistant-SFT-7-Llama-30B-GPTQ.
Click Download.
Wait until it says it's finished downloading.
Click the Refresh icon next to Model in the top left.
In the Model drop-down: choose the model you just downloaded, OpenAssistant-SFT-7-Llama-30B-GPTQ.
If you see an error in the bottom right, ignore it - it's temporary.
Fill out the GPTQ parameters on the right: Bits = 4, Groupsize = None, model_type = Llama
Click Save settings for this model in the top right.
Click Reload the Model in the top right.
Once it says it's loaded, click the Text Generation tab and enter a prompt!

Manual instructions for `text-generation-webui`

The compat.no-act-order.safetensors files can be loaded the same as any other GPTQ file, without requiring any updates to oobaboogas text-generation-webui.

Instructions on using GPTQ 4bit files in text-generation-webui are here.

The latest.act-order.safetensors files were created using --act-order to give the maximum possible quantisation quality, but this means it requires that the latest GPTQ-for-LLaMa is used inside the UI.

If you want to use the act-order safetensors files and need to update the Triton branch of GPTQ-for-LLaMa, here are the commands I used to clone the Triton branch of GPTQ-for-LLaMa, clone text-generation-webui, and install GPTQ into the UI:

# Clone text-generation-webui, if you don't already have it
git clone https://github.com/oobabooga/text-generation-webui
# Make a repositories directory
mkdir text-generation-webui/repositories
cd text-generation-webui/repositories
# Clone the latest GPTQ-for-LLaMa code inside text-generation-webui
git clone https://github.com/qwopqwop200/GPTQ-for-LLaMa

Then install this model into text-generation-webui/models and launch the UI as follows:

cd text-generation-webui
python server.py --model OpenAssistant-SFT-7-Llama-30B-GPTQ --wbits 4 --groupsize 128 --model_type Llama # add any other command line args you want

To update the CUDA branch of GPTQ-for-LLaMa, you can do the following. This requires a C/C++ compiler and the CUDA toolkit installed!

# Clone text-generation-webui, if you don't already have it
git clone https://github.com/oobabooga/text-generation-webui
# Make a repositories directory
mkdir text-generation-webui/repositories
cd text-generation-webui/repositories
# Clone the latest GPTQ-for-LLaMa code inside text-generation-webui
git clone -b cuda https://github.com/qwopqwop200/GPTQ-for-LLaMa
cd GPTQ-for-LLaMa
pip uninstall quant-cuda # uninstall existing CUDA version
python setup_cuda.py install # install latest version

The above commands assume you have installed all dependencies for GPTQ-for-LLaMa and text-generation-webui. Please see their respective repositories for further information.

If you can't update GPTQ-for-LLaMa or don't want to, please use a compat.no-act-order.safetensor file.

Original model card

llama-30b-sft-7:
  dtype: fp16
  log_dir: "llama_log_30b"
  learning_rate: 1e-5
  model_name: /home/ubuntu/Open-Assistant/model/model_training/.saved/llama-30b-super-pretrain/checkpoint-3500
  #model_name: OpenAssistant/llama-30b-super-pretrain
  output_dir: llama_model_30b
  deepspeed_config: configs/zero3_config_sft.json
  weight_decay: 0.0
  residual_dropout: 0.0
  max_length: 2048
  use_flash_attention: true
  warmup_steps: 20
  gradient_checkpointing: true
  gradient_accumulation_steps: 12
  per_device_train_batch_size: 2
  per_device_eval_batch_size: 3
  eval_steps: 101
  save_steps: 485
  num_train_epochs: 4
  save_total_limit: 3
  use_custom_sampler: true
  sort_by_length: false
  #save_strategy: steps
  save_strategy: epoch
  datasets:
    - oasst_export:
        lang: "bg,ca,cs,da,de,en,es,fr,hr,hu,it,nl,pl,pt,ro,ru,sl,sr,sv,uk"
        input_file_path: 2023-04-12_oasst_release_ready_synth.jsonl.gz
        val_split: 0.05
    - vicuna:
        val_split: 0.05
        max_val_set: 800
        fraction: 1.0
    - dolly15k:
        val_split: 0.05
        max_val_set: 300
    - grade_school_math_instructions:
        val_split: 0.05
    - code_alpaca:
        val_split: 0.05
        max_val_set: 250

OASST dataset paper: https://arxiv.org/abs/2304.07327