Update README.md
Browse files
README.md
CHANGED
|
@@ -3,4 +3,17 @@ license: apache-2.0
|
|
| 3 |
datasets:
|
| 4 |
- ssmi153/Capybara-ShareGPT
|
| 5 |
- jondurbin/airoboros-3.2
|
| 6 |
-
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
datasets:
|
| 4 |
- ssmi153/Capybara-ShareGPT
|
| 5 |
- jondurbin/airoboros-3.2
|
| 6 |
+
---
|
| 7 |
+
|
| 8 |
+
QLoRA fine-tune of [Mixtral-8x22B-v0.1](https://huggingface.co/mistral-community/Mixtral-8x22B-v0.1) on a combination of the Capybara and Airoboros datasets.
|
| 9 |
+
|
| 10 |
+
Uses Mistral instruct formatting, like this:
|
| 11 |
+
[INST] Describe quantum computing to a layperson. [/INST]
|
| 12 |
+
|
| 13 |
+
Model details:
|
| 14 |
+
- Trained with QLoRA, on 4 4090s, using my own [qlora-pipe](https://github.com/tdrussell/qlora-pipe) training script
|
| 15 |
+
- LoRA rank 64
|
| 16 |
+
- 4096 sequence length
|
| 17 |
+
- 2 epochs
|
| 18 |
+
|
| 19 |
+
You can find the LoRA adapter files [here](https://huggingface.co/tdrussell/Mixtral-8x22B-Capyboros-v1-lora). I have also uploaded a single quant (GGUF q4_k_s) [here](https://huggingface.co/tdrussell/Mixtral-8x22B-Capyboros-v1-GGUF-q4_k_s) if you want to try it without quantizing yourself or waiting for someone else to make all the quants. It fits with at least 16k context length on 96GB VRAM.
|