Spaces:
Running
Running
birgermoell
commited on
Commit
β’
9e416c9
1
Parent(s):
11e234f
Update README.md
Browse files
README.md
CHANGED
@@ -7,4 +7,45 @@ sdk: static
|
|
7 |
pinned: false
|
8 |
---
|
9 |
|
10 |
-
The merge crew is the mergiest crew.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
7 |
pinned: false
|
8 |
---
|
9 |
|
10 |
+
The merge crew is the mergiest crew. π
|
11 |
+
|
12 |
+
## Merge Crew Planning Document
|
13 |
+
https://docs.google.com/document/d/1fP2FIrCifWcLGdTBmqeogdCdZJOwxqPfEyO-HA76_qc/edit?usp=sharing
|
14 |
+
|
15 |
+
## Merging tutorial
|
16 |
+
https://huggingface.co/blog/mlabonne/merge-models
|
17 |
+
|
18 |
+
## Colab for merging
|
19 |
+
Lazy merge kit notebook for merging models.
|
20 |
+
https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing
|
21 |
+
|
22 |
+
## Model Merging Discord
|
23 |
+
Discord channel for discussions of model mergin.
|
24 |
+
https://discord.com/channels/905500133343518791/1202582325146161183
|
25 |
+
|
26 |
+
|
27 |
+
## Merge methods
|
28 |
+
When working you use a YAML file like the following
|
29 |
+
|
30 |
+
```
|
31 |
+
models:
|
32 |
+
- model: timpal0l/BeagleCatMunin
|
33 |
+
# No parameters necessary for base model
|
34 |
+
- model: bineric/NorskGPT-Mistral-7b
|
35 |
+
parameters:
|
36 |
+
density: 0.53
|
37 |
+
weight: 0.6
|
38 |
+
merge_method: dare_ties
|
39 |
+
base_model: timpal0l/BeagleCatMunin
|
40 |
+
parameters:
|
41 |
+
int8_mask: true
|
42 |
+
dtype: bfloat16
|
43 |
+
```
|
44 |
+
|
45 |
+
The dare_ties methods seems to perform better than other merging methods.
|
46 |
+
Learn more about merge ties here.
|
47 |
+
https://arxiv.org/pdf/2306.01708.pdf
|
48 |
+
|
49 |
+
|
50 |
+
|
51 |
+
|