---
license: other
inference: false
datasets:
- gozfarb/ShareGPT_Vicuna_unfiltered
---

# VicUnlocked-30B-LoRA GPTQ

This is GPTQ format quantised 4bit models of [Neko Institute of Science's VicUnLocked 30B LoRA](https://huggingface.co/Neko-Institute-of-Science/VicUnLocked-30b-LoRA).

The files in this repo are the result of merging the above LoRA with the original LLaMA 30B, then quantising to 4bit using [GPTQ-for-LLaMa](https://github.com/qwopqwop200/GPTQ-for-LLaMa).

## Repositories available

* [4-bit, 5-bit and 8-bit GGML models for CPU inference](https://huggingface.co/TheBloke/VicUnlocked-30B-LoRA-GGML).
* [4bit's GPTQ 4-bit model for GPU inference](https://huggingface.co/TheBloke/VicUnlocked-30B-LoRA-GPTQ).
* [float16 HF format model for GPU inference and further conversions](https://huggingface.co/TheBloke/VicUnlocked-30B-LoRA-HF).

## How to easily download and use this model in text-generation-webui

Open the text-generation-webui UI as normal.

1. Click the **Model tab**.
2. Under **Download custom model or LoRA**, enter `TheBloke/VicUnlocked-30B-LoRA-GPTQ`.
3. Click **Download**.
4. Wait until it says it's finished downloading.
5. Click the **Refresh** icon next to **Model** in the top left.
6. In the **Model drop-down**: choose the model you just downloaded, `VicUnlocked-30B-LoRA-GPTQ`.
7. If you see an error in the bottom right, ignore it - it's temporary.
8. Fill out the `GPTQ parameters` on the right: `Bits = 4`, `Groupsize = None`, `model_type = Llama`
9. Click **Save settings for this model** in the top right.
10. Click **Reload the Model** in the top right.
11. Once it says it's loaded, click the **Text Generation tab** and enter a prompt!

## Provided files

**Compatible file - VicUnlocked-30B-LoRA-GPTQ-4bit.act-order.safetensors**

In the `main` branch - the default one - you will find `VicUnlocked-30B-LoRA-GPTQ-4bit-128g.compat.no-act-order.safetensors`

This will work with all versions of GPTQ-for-LLaMa. It has maximum compatibility

It was created without groupsize so as to minimise VRAM requirements. It is created with the `--act-order` parameter to improve inference quality. 

* `VicUnlocked-30B-LoRA-GPTQ-4bit-128g.compat.no-act-order.safetensors`
  * Works with all versions of GPTQ-for-LLaMa code, both Triton and CUDA branches
  * Works with AutoGPTQ.
  * Works with text-generation-webui one-click-installers
  * Parameters: Groupsize = None. act-order.
  * Command used to create the GPTQ:
    ```
    llama.py /workspace/vicunlocked-30b/HF wikitext2 --wbits 4 --true-sequential --act-order   --save_safetensors /workspace/vicunlocked-30b/gptq/VicUnlocked-30B-GPTQ-4bit.act-order.safetensors
    ```


# Original model card

# Convert tools
https://github.com/practicaldreamer/vicuna_to_alpaca

# Training tool
https://github.com/oobabooga/text-generation-webui

ATM I'm using 2023.05.04v0 of the dataset and training full context.

# Notes:
So I will only be training 1 epoch, as full context 30b takes so long to train.
This 1 epoch will take me 8 days lol but luckily these LoRA feels fully functinal at epoch 1 as shown on my 13b one.
Also I will be uploading checkpoints almost everyday. I could train another epoch if there's enough want for it.

Update: Since I will not be training over 1 epoch @Aeala is training for the full 3 https://huggingface.co/Aeala/VicUnlocked-alpaca-half-30b-LoRA but it's half ctx if you care about that. Also @Aeala's just about done.

Update: Training Finished at Epoch 1, These 8 days sure felt long. I only have one A6000 lads there's only so much I can do. Also RIP gozfarb IDK what happened to him.

# How to test?
1. Download LLaMA-30B-HF if you have not: https://huggingface.co/Neko-Institute-of-Science/LLaMA-30B-HF
2. Make a folder called VicUnLocked-30b-LoRA in the loras folder.
3. Download adapter_config.json and adapter_model.bin into VicUnLocked-30b-LoRA.
4. Load ooba: ```python server.py --listen --model LLaMA-30B-HF --load-in-8bit --chat --lora VicUnLocked-30b-LoRA```
5. Select instruct and chose Vicuna-v1.1 template.


# Training Log
https://wandb.ai/neko-science/VicUnLocked/runs/vx8yzwi7