Commit
•
8de079d
1
Parent(s):
93eb9da
Update README.md
Browse files
README.md
CHANGED
|
@@ -13,8 +13,11 @@ tags:
|
|
| 13 |
|
| 14 |
# Lumosia-MoE-4x10.7
|
| 15 |
|
|
|
|
|
|
|
| 16 |
This is a very experimantal model. its a MoE of all good performing Solar models (based off of personal experiance not open leaderboard),
|
| 17 |
-
|
|
|
|
| 18 |
|
| 19 |
context is maybe 32k? waiting for GGUF to upload.
|
| 20 |
|
|
@@ -42,7 +45,8 @@ Lumosia-MoE-4x10.7 is a Mixure of Experts (MoE) made with the following models:
|
|
| 42 |
|
| 43 |
## 🧩 Configuration
|
| 44 |
|
| 45 |
-
```
|
|
|
|
| 46 |
gate_mode: hidden
|
| 47 |
dtype: bfloat16
|
| 48 |
experts:
|
|
@@ -53,11 +57,13 @@ experts:
|
|
| 53 |
- source_model: kyujinpy/Sakura-SOLAR-Instruct
|
| 54 |
positive_prompts: [""]
|
| 55 |
- source_model: jeonsworld/CarbonVillain-en-10.7B-v1
|
| 56 |
-
positive_prompts: [""]
|
|
|
|
| 57 |
|
| 58 |
## 💻 Usage
|
| 59 |
|
| 60 |
-
```
|
|
|
|
| 61 |
!pip install -qU transformers bitsandbytes accelerate
|
| 62 |
|
| 63 |
from transformers import AutoTokenizer
|
|
|
|
| 13 |
|
| 14 |
# Lumosia-MoE-4x10.7
|
| 15 |
|
| 16 |
+
The name Lumosia was selected as its a MoE of Multiple SOLAR Merges so it really "Lights the way".... its 3am.
|
| 17 |
+
|
| 18 |
This is a very experimantal model. its a MoE of all good performing Solar models (based off of personal experiance not open leaderboard),
|
| 19 |
+
|
| 20 |
+
Why? Dunno whated to see what would happen
|
| 21 |
|
| 22 |
context is maybe 32k? waiting for GGUF to upload.
|
| 23 |
|
|
|
|
| 45 |
|
| 46 |
## 🧩 Configuration
|
| 47 |
|
| 48 |
+
```
|
| 49 |
+
yamlbase_model: DopeorNope/SOLARC-M-10.7B
|
| 50 |
gate_mode: hidden
|
| 51 |
dtype: bfloat16
|
| 52 |
experts:
|
|
|
|
| 57 |
- source_model: kyujinpy/Sakura-SOLAR-Instruct
|
| 58 |
positive_prompts: [""]
|
| 59 |
- source_model: jeonsworld/CarbonVillain-en-10.7B-v1
|
| 60 |
+
positive_prompts: [""]
|
| 61 |
+
```
|
| 62 |
|
| 63 |
## 💻 Usage
|
| 64 |
|
| 65 |
+
```
|
| 66 |
+
python
|
| 67 |
!pip install -qU transformers bitsandbytes accelerate
|
| 68 |
|
| 69 |
from transformers import AutoTokenizer
|