birgermoell commited on
Commit
9e416c9
β€’
1 Parent(s): 11e234f

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +42 -1
README.md CHANGED
@@ -7,4 +7,45 @@ sdk: static
7
  pinned: false
8
  ---
9
 
10
- The merge crew is the mergiest crew. My papa had a saying, when in doubt, merge. 😎
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
7
  pinned: false
8
  ---
9
 
10
+ The merge crew is the mergiest crew. 😎
11
+
12
+ ## Merge Crew Planning Document
13
+ https://docs.google.com/document/d/1fP2FIrCifWcLGdTBmqeogdCdZJOwxqPfEyO-HA76_qc/edit?usp=sharing
14
+
15
+ ## Merging tutorial
16
+ https://huggingface.co/blog/mlabonne/merge-models
17
+
18
+ ## Colab for merging
19
+ Lazy merge kit notebook for merging models.
20
+ https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing
21
+
22
+ ## Model Merging Discord
23
+ Discord channel for discussions of model mergin.
24
+ https://discord.com/channels/905500133343518791/1202582325146161183
25
+
26
+
27
+ ## Merge methods
28
+ When working you use a YAML file like the following
29
+
30
+ ```
31
+ models:
32
+ - model: timpal0l/BeagleCatMunin
33
+ # No parameters necessary for base model
34
+ - model: bineric/NorskGPT-Mistral-7b
35
+ parameters:
36
+ density: 0.53
37
+ weight: 0.6
38
+ merge_method: dare_ties
39
+ base_model: timpal0l/BeagleCatMunin
40
+ parameters:
41
+ int8_mask: true
42
+ dtype: bfloat16
43
+ ```
44
+
45
+ The dare_ties methods seems to perform better than other merging methods.
46
+ Learn more about merge ties here.
47
+ https://arxiv.org/pdf/2306.01708.pdf
48
+
49
+
50
+
51
+