File size: 4,327 Bytes
238c5da |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 |
---
base_model:
- unsloth/Mistral-Small-Instruct-2409
- TheDrummer/Cydonia-22B-v1.2
- Gryphe/Pantheon-RP-Pure-1.6.2-22b-Small
- anthracite-org/magnum-v4-22b
- ArliAI/Mistral-Small-22B-ArliAI-RPMax-v1.1
- spow12/ChatWaifu_v2.0_22B
- rAIfle/Acolyte-22B
- Envoid/Mistral-Small-NovusKyver
- InferenceIllusionist/SorcererLM-22B
library_name: transformers
tags:
- mergekit
- merge
license: other
language:
- en
---
***
## Schisandra
Many thanks to the authors of the models used!
[RPMax v1.1](https://huggingface.co/ArliAI/Mistral-Small-22B-ArliAI-RPMax-v1.1) | [Pantheon-RP](https://huggingface.co/Gryphe/Pantheon-RP-Pure-1.6.2-22b-Small) | [Cydonia v1.2](https://huggingface.co/TheDrummer/Cydonia-22B-v1.2) | [Magnum V4](https://huggingface.co/anthracite-org/magnum-v4-22b) | [ChatWaifu v2.0](https://huggingface.co/spow12/ChatWaifu_v2.0_22B) | [SorcererLM](https://huggingface.co/InferenceIllusionist/SorcererLM-22B) | [Acolyte](https://huggingface.co/rAIfle/Acolyte-22B) | [NovusKyver](https://huggingface.co/Envoid/Mistral-Small-NovusKyver)
***
### Overview
Main uses: RP, Storywriting
Merge of 8 Mistral Small finetunes in total, which were then merged back into the original model to make it less stupid. Worked somehow? Definitely smarter than my previous MS merge and maybe some finetunes. Seems to really adhere to the writing style of the previous output, so you'll need either a good character card or an existing chat for a better replies.
***
### Quants
[Static](https://huggingface.co/mradermacher/MS-Schisandra-22B-vB-GGUF)
[Imatrix](https://huggingface.co/mradermacher/MS-Schisandra-22B-vB-i1-GGUF)
***
### Settings
Prompt format: Mistral-V3 Tekken
Samplers: [These](https://qu.ax/OusTx.json) or [These](https://huggingface.co/ToastyPigeon/ST-Presets-Mistral-Small/resolve/main/ST-sampling-preset-Mistral-Small.json?download=true)
***
## Merge Details
### Merging steps
## QCmix
```yaml
base_model: InferenceIllusionist/SorcererLM-22B
parameters:
int8_mask: true
rescale: true
normalize: false
dtype: bfloat16
tokenizer_source: base
merge_method: della
models:
- model: Envoid/Mistral-Small-NovusKyver
parameters:
density: [0.35, 0.65, 0.5, 0.65, 0.35]
epsilon: [0.1, 0.1, 0.25, 0.1, 0.1]
lambda: 0.85
weight: [-0.01891, 0.01554, -0.01325, 0.01791, -0.01458]
- model: rAIfle/Acolyte-22B
parameters:
density: [0.6, 0.4, 0.5, 0.4, 0.6]
epsilon: [0.15, 0.15, 0.25, 0.15, 0.15]
lambda: 0.85
weight: [0.01768, -0.01675, 0.01285, -0.01696, 0.01421]
```
## Schisandra-vA
```yaml
merge_method: della_linear
dtype: bfloat16
parameters:
normalize: true
int8_mask: true
tokenizer_source: union
base_model: TheDrummer/Cydonia-22B-v1.2
models:
- model: ArliAI/Mistral-Small-22B-ArliAI-RPMax-v1.1
parameters:
density: 0.55
weight: 1
- model: Gryphe/Pantheon-RP-Pure-1.6.2-22b-Small
parameters:
density: 0.55
weight: 1
- model: spow12/ChatWaifu_v2.0_22B
parameters:
density: 0.55
weight: 1
- model: anthracite-org/magnum-v4-22b
parameters:
density: 0.55
weight: 1
- model: QCmix
parameters:
density: 0.55
weight: 1
```
## Schisandra
```yaml
dtype: bfloat16
tokenizer_source: base
merge_method: della_linear
parameters:
density: 0.5
base_model: Schisandra
models:
- model: unsloth/Mistral-Small-Instruct-2409
parameters:
weight:
- filter: v_proj
value: [0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0]
- filter: o_proj
value: [1, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1]
- filter: up_proj
value: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
- filter: gate_proj
value: [0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0]
- filter: down_proj
value: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
- value: 0
- model: Schisandra
parameters:
weight:
- filter: v_proj
value: [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1]
- filter: o_proj
value: [0, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0]
- filter: up_proj
value: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
- filter: gate_proj
value: [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1]
- filter: down_proj
value: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
- value: 1
``` |