---
license: llama3
tags:
- moe
language:
- en
---

![image/png](https://cdn-uploads.huggingface.co/production/uploads/64f5e51289c121cb864ba464/K3jwX1gq_nohz7Fe78xev.png)
(Maybe i'll change the waifu picture later.)

Experimental RP-oriented MoE, the idea was to get a model that would be equal to or better than the Mixtral 8x7B and it's finetunes in RP/ERP tasks.

The model has totally 25B parameters, of which ~13B are active.

Please feedback me if it's more stable than the [previous version](https://huggingface.co/xxx777xxxASD/L3-ChaoticSoliloquy-v1.5-4x8B)

### Llama 3 ChaoticSoliloquy-v2-4x8B test
```
base_model: L3_ChaosMaid_8B
gate_mode: random
dtype: bfloat16
experts_per_token: 2
experts:
  - source_model: ChaoticNeutrals_Poppy_Porpoise-0.72-L3-8B
  - source_model: L3_ChaosMaid_8B
  - source_model: openlynn_Llama-3-Soliloquy-8B-v2
  - source_model: Sao10K_L3-Solana-8B-v1
```


## Models used

- [ChaoticNeutrals/Poppy_Porpoise-0.72-L3-8B](https://huggingface.co/ChaoticNeutrals/Poppy_Porpoise-0.72-L3-8B)
- [jeiku/Chaos_RP_l3_8B](https://huggingface.co/jeiku/Chaos_RP_l3_8B)
- [NeverSleep/Llama-3-Lumimaid-8B-v0.1](https://huggingface.co/NeverSleep/Llama-3-Lumimaid-8B-v0.1)
- [openlynn/Llama-3-Soliloquy-8B-v2](https://huggingface.co/openlynn/Llama-3-Soliloquy-8B-v2)
- [Sao10K/L3-Solana-8B-v1](https://huggingface.co/Sao10K/L3-Solana-8B-v1)


## Difference

- Update from [ChaoticNeutrals/Poppy_Porpoise-v0.7-L3-8B](https://huggingface.co/ChaoticNeutrals/Poppy_Porpoise-v0.7-L3-8B) to [ChaoticNeutrals/Poppy_Porpoise-0.72-L3-8B](https://huggingface.co/ChaoticNeutrals/Poppy_Porpoise-0.72-L3-8B)
- Update from [openlynn/Llama-3-Soliloquy-8B](https://huggingface.co/openlynn/Llama-3-Soliloquy-8B) to [openlynn/Llama-3-Soliloquy-8B-v2](https://huggingface.co/openlynn/Llama-3-Soliloquy-8B-v2)
- Change - [NeverSleep/Llama-3-Lumimaid-8B-v0.1](https://huggingface.co/NeverSleep/Llama-3-Lumimaid-8B-v0.1) to L3-ChaosMaid-8B

## L3 ChaosMaid-8B
```
models:
  - model: jeiku_Chaos_RP_l3_8B
    # No parameters necessary for base model
  - model: NeverSleep_Llama-3-Lumimaid-8B-v0.1
    parameters:
      density: 0.5
      weight: 0.5
merge_method: dare_ties
base_model: jeiku_Chaos_RP_l3_8B
parameters:
  int8_mask: true
dtype: bfloat16
```

## Vision

[llama3_mmproj](https://huggingface.co/ChaoticNeutrals/LLaVA-Llama-3-8B-mmproj-Updated)

![image/png](https://cdn-uploads.huggingface.co/production/uploads/64f5e51289c121cb864ba464/yv4C6NalqORLjvY3KKZk8.png)


## Prompt format: Llama 3