Hugging Face's logo

Spaces:
merge-crew
/
README
Running

App Files Community
README
File size: 1,189 Bytes 
74f9391
 
 
 
 
 
 
 
 
9e416c9
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
35921f3
9e416c9
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
 
---
title: README
emoji: 🐠
colorFrom: indigo
colorTo: red
sdk: static
pinned: false
---

The merge crew is the mergiest crew. 😎

## Merge Crew Planning Document
https://docs.google.com/document/d/1fP2FIrCifWcLGdTBmqeogdCdZJOwxqPfEyO-HA76_qc/edit?usp=sharing

## Merging tutorial
https://huggingface.co/blog/mlabonne/merge-models

## Colab for merging
Lazy merge kit notebook for merging models.
https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing

## Model Merging Discord
Discord channel for discussions of model mergin.
https://discord.com/channels/905500133343518791/1202582325146161183


## Merge methods
When working you use a YAML file like the following

```
  models:
    - model: timpal0l/BeagleCatMunin
      # No parameters necessary for base model
    - model: bineric/NorskGPT-Mistral-7b
      parameters:
        density: 0.53
        weight: 0.6
  merge_method: dare_ties
  base_model: timpal0l/BeagleCatMunin
  parameters:
    int8_mask: true
  dtype: bfloat16
  random_seed: 42
```

The dare_ties methods seems to perform better than other merging methods.
Learn more about merge ties here.
https://arxiv.org/pdf/2306.01708.pdf