---
tags:
- not-for-all-audiences
- nsfw
license: other
language:
- en
---

[EXL2](https://github.com/turboderp/exllamav2/tree/master#exllamav2) Quantization of [Undi95's's MXLewd-L2-20B](https://huggingface.co/Undi95/MXLewd-L2-20B).


## Model details

First attempt to quantize a 20B model so it can run on 16GB VRAM with the highest quality possible.
Quantized at 3.18bpw with hb 6. 8.13bpw also available for those who want it (exl2 is very fast with flash-attention and the quality is (almost) the same with fp16.)

Perplexity:

Base = 6.4744

8bpw h8 = 6.4471

3.18 h6 = 6.5705

Dataset = [wikitext](https://huggingface.co/datasets/wikitext/resolve/refs%2Fconvert%2Fparquet/wikitext-2-v1/test/0000.parquet)

## Prompt Format

```
Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{prompt}

### Response:

```