|
--- |
|
|
|
|
|
{} |
|
--- |
|
|
|
# Model Card for Model ID |
|
|
|
<!-- Provide a quick summary of what the model is/does. --> |
|
|
|
This model is an extreme experiment, I wanted to test making an MoE out of multiple High Performing Solar models. Let me know what you think |
|
|
|
## Model Details |
|
|
|
### Model Description |
|
|
|
This MoE model is an extreme experiment, I wanted to test making an MoE out of multiple High Performing Solar models. Let me know what you think. |
|
Thinking about finetuning on a RP dataset later on to direct the model more |
|
|
|
- **Model type:** [More Information Needed] |
|
|
|
### Model Sources [optional] |
|
|
|
model_name: Lumosia-MoE-4x10.7 |
|
base_model: DopeorNope/SOLARC-M-10.7B |
|
gate_mode: hidden |
|
dtype: bfloat16 |
|
experts: |
|
- source_model: DopeorNope/SOLARC-M-10.7B |
|
positive_prompts: [""] |
|
- source_model: maywell/PiVoT-10.7B-Mistral-v0.2-RP |
|
positive_prompts: [""] |
|
- source_model: kyujinpy/Sakura-SOLAR-Instruct |
|
positive_prompts: [""] |
|
- source_model: jeonsworld/CarbonVillain-en-10.7B-v1 |
|
positive_prompts: [""] |
|
|
|
|