Hugging Face's logo

SteelStorage
/
Lumosia-MoE-4x10.7

Text Generation
Transformers
Safetensors
mixtral
Mixture of Experts
Merge
mergekit
Solar Moe
Solar
Lumosia
conversational
Eval Results
text-generation-inference
Inference Endpoints
Model card Files Community
1
Lumosia-MoE-4x10.7 / README.md
Steelskull's picture
Steelskull
Create README.md
07e18df
preview code
|
raw
history blame
No virus
1.19 kB
metadata
{}

Model Card for Model ID

This model is an extreme experiment, I wanted to test making an MoE out of multiple High Performing Solar models. Let me know what you think

Model Details

Model Description

This MoE model is an extreme experiment, I wanted to test making an MoE out of multiple High Performing Solar models. Let me know what you think. Thinking about finetuning on a RP dataset later on to direct the model more

  • Model type: [More Information Needed]

Model Sources [optional]

model_name: Lumosia-MoE-4x10.7 base_model: DopeorNope/SOLARC-M-10.7B gate_mode: hidden dtype: bfloat16 experts:

  • source_model: DopeorNope/SOLARC-M-10.7B positive_prompts: [""]
  • source_model: maywell/PiVoT-10.7B-Mistral-v0.2-RP positive_prompts: [""]
  • source_model: kyujinpy/Sakura-SOLAR-Instruct positive_prompts: [""]
  • source_model: jeonsworld/CarbonVillain-en-10.7B-v1 positive_prompts: [""]