---
base_model:
- mlabonne/Hermes-3-Llama-3.1-8B-lorablated
license: llama3
tags:
- mergekit
- merge

---
# Hermes-3-Llama-3.1-8B-lorablated-exl2
Model: [Hermes-3-Llama-3.1-8B-lorablated](https://huggingface.co/mlabonne/Hermes-3-Llama-3.1-8B-lorablated)  
Created by: [mlabonne](https://huggingface.co/mlabonne)  
Based on: [Hermes-3-Llama-3.1-8B](https://huggingface.co/NousResearch/Hermes-3-Llama-3.1-8B)

## Quants
[4bpw h6](https://huggingface.co/cgus/Hermes-3-Llama-3.1-8B-lorablated-exl2/tree/main)  
[4.5bpw h6](https://huggingface.co/cgus/Hermes-3-Llama-3.1-8B-lorablated-exl2/tree/4.5bpw-h6)  
[5bpw h6](https://huggingface.co/cgus/Hermes-3-Llama-3.1-8B-lorablated-exl2/tree/5bpw-h6)  
[6bpw h6](https://huggingface.co/cgus/Hermes-3-Llama-3.1-8B-lorablated-exl2/tree/6bpw-h6)  
[8bpw h8](https://huggingface.co/cgus/Hermes-3-Llama-3.1-8B-lorablated-exl2/tree/8bpw-h8)  

## Quantization notes
Made with Exllamav2 0.1.8 with the default dataset.  
I'm not sure how well it works with Text-Generation-WebUI considering that this model uses some unusual RoPE mechanics and I have no idea how TGW handles it.  
For some reason this model worked extremely slow with my TGW install but was perfectly fine with TabbyAPI.

## How to run
I recommend using TabbyAPI for this model. The model requires a decent Nvidia RTX card on Windows/Linux or a decent AMD GPU on Linux.  
It requires to be fully loaded in GPU to work, so if your GPU has too small VRAM you should use [GGUF version](https://huggingface.co/mlabonne/Hermes-3-Llama-3.1-8B-lorablated-GGUF) instead.  
If you have Nvidia GTX card you should also use GGUF instead.

# Orignal model card
# Hermes-3-Llama-3.1-8B-lorablated

![image/png](https://cdn-uploads.huggingface.co/production/uploads/61b8e2ba285851687028d395/4Hbw5n68jKUSBQeTqQIeT.png)
<center>70B version: <a href="https://huggingface.co/mlabonne/Hermes-3-Llama-3.1-70B-lorablated/"><i>mlabonne/Hermes-3-Llama-3.1-70B-lorablated</i></a></center>

This is an uncensored version of [NousResearch/Hermes-3-Llama-3.1-8B](https://huggingface.co/NousResearch/Hermes-3-Llama-3.1-8B) using lorablation.

You can see in the following example how Hermes 3 refuses to answer a legitimate question while the abliterated model complies:

![image/png](https://cdn-uploads.huggingface.co/production/uploads/61b8e2ba285851687028d395/2-ZRBvlZxvIr_Ag_ynNkk.png)

The recipe is based on @grimjim's [grimjim/Llama-3.1-8B-Instruct-abliterated_via_adapter](https://huggingface.co/grimjim/Llama-3.1-8B-Instruct-abliterated_via_adapter) (special thanks):

1. **Extraction**: We extract a LoRA adapter by comparing two models: a censored Llama 3.1 ([meta-llama/Meta-Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct)) and an abliterated Llama 3.1 ([mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated](https://huggingface.co/mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated)).
2. **Merge**: We merge this new LoRA adapter using [task arithmetic](https://arxiv.org/abs/2212.04089) to the censored [NousResearch/Hermes-3-Llama-3.1-8B](https://huggingface.co/NousResearch/Hermes-3-Llama-3.1-8B) to abliterate it.

![image/png](https://cdn-uploads.huggingface.co/production/uploads/61b8e2ba285851687028d395/JdYyK-HLHbyBiHvg-Nvsn.png)

See [this article](https://huggingface.co/blog/mlabonne/abliteration) to learn more about abliteration.

## ⚡ Quantization

* **GGUF**: https://huggingface.co/mlabonne/Hermes-3-Llama-3.1-8B-lorablated-GGUF

## 🧩 Configuration

This model was merged using the [task arithmetic](https://arxiv.org/abs/2212.04089) merge method using [NousResearch/Hermes-3-Llama-3.1-8B](https://huggingface.co/NousResearch/Hermes-3-Llama-3.1-8B) + Llama-3.1-8B-Instruct-abliterated-LORA as a base.

The following YAML configuration was used to produce this model:

```yaml
base_model: NousResearch/Hermes-3-Llama-3.1-8B+Llama-3.1-8B-Instruct-abliterated-LORA
dtype: bfloat16
merge_method: task_arithmetic
parameters:
  normalize: false
slices:
- sources:
  - layer_range: [0, 32]
    model: NousResearch/Hermes-3-Llama-3.1-8B+Llama-3.1-8B-Instruct-abliterated-LORA
    parameters:
      weight: 1.0
```

You can reproduce this model using the following commands:

```bash
# Setup
git clone https://github.com/arcee-ai/mergekit.git
cd mergekit && pip install -e .
pip install bitsandbytes

# Extraction
mergekit-extract-lora mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated meta-llama/Meta-Llama-3.1-8B-Instruct Llama-3.1-8B-Instruct-abliterated-LORA --rank=64

# Merge using previous config
mergekit-yaml config.yaml Hermes-3-Llama-3.1-8B-lorablated --allow-crimes --lora-merge-cache=./cache
```