Hugging Face's logo

HiroseKoichi
/
Llama-Salad-4x8B-V2

Text Generation
Transformers
Safetensors
mixtral
nsfw
Not-For-All-Audiences
llama-3
text-generation-inference
Mixture of Experts
mergekit
Merge
conversational
Inference Endpoints
Model card Files Community
1
Llama-Salad-4x8B-V2 / README.md
HiroseKoichi's picture
HiroseKoichi
Update README.md
069cf1d verified
preview code
|
raw
history blame
2.97 kB
metadata
license: llama3
library_name: transformers
tags:
  - nsfw
  - not-for-all-audiences
  - llama-3
  - text-generation-inference

Llama-Salad-4x8B-V2

Changes in V2:

  • Swapped Tess-2.0-Llama-3-8B for Llama-3-8B-Synthia-v3.5
  • Swapped L3-8B-Stheno-v3.1 for Llama-3-Soliloquy-8B-v2
  • Removed Llama3-OpenBioLLM-8B and added opus-v1.2-llama-3-8b-instruct-run3.5-epoch2.5

V2 has improvements in all areas from V1; it's not a massive improvement, but I can confidently say it's a direct upgrade. Llama-3-8B-Synthia-v3.5 is better than Tess-2.0-Llama-3-8B in every way; Llama-3-Soliloquy-8B-v2 is more intelligent than L3-8B-Stheno-v3.1 and has less bias towards NSFW content; and the inclusion of opus-v1.2-llama-3-8b-instruct-run3.5-epoch2.5 has greatly improved its storytelling and narration abilities.

I really like the model selection in this one, so I don't know how much more I can improve if I make another 4x8B merge. If I were to make a V3, swapping Meta-Llama-3-8B-Instruct would likely be the only change. I will try my hand at making an 8x8B merge in the future, but I still need to find some models to fill the gaps; making sure there's no routing conflicts between 8 different models at once will be the biggest challenge.

Details

Models Used

Merge Config 

base_model: NousResearch/Meta-Llama-3-8B-Instruct
gate_mode: hidden
dtype: bfloat16
experts_per_token: 2
experts:
  - source_model: NousResearch/Meta-Llama-3-8B-Instruct
    positive_prompts:
    - "summarize"
    - "paraphrase"
    - "explain"
    - "define"
    - "translate"
    - "multilingual"
    - "chat"
    - "conversation"
  - source_model: migtissera/Llama-3-8B-Synthia-v3.5
    positive_prompts:
    - "programming language"
    - "JavaScript"
    - "Python programming language"
    - "Rust programming language"
    - "CSS markup styling language"
    - "math"
    - "code"
    - "step-by-step"
    - "logical reasoning"
  - source_model: openlynn/Llama-3-Soliloquy-8B-v2
    positive_prompts:
    - "roleplay"
    - "erotic roleplay"
    - "characters"
    - "scene"
    - "opinion"
  - source_model: dreamgen-preview/opus-v1.2-llama-3-8b-instruct-run3.5-epoch2.5
    positive_prompts:
    - "creative writing"
    - "storytelling"
    - "narration"
    - "narrative setting"
    - "narrative plot"
    - "narrative exposition"
    - "narrative theme"
    - "narrative climax"