saattrupdan commited on
Commit
144039c
·
verified ·
1 Parent(s): 17f3015

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +5 -16
README.md CHANGED
@@ -19,11 +19,10 @@ base_model:
19
 
20
  # ScandiMerge
21
 
22
- This is a DARE-TIES merge of the following models, all based on `mistralai/Mistral-7B-v0.1`:
23
 
24
  1. `danish-foundation-models/munin-7b-alpha`, continued pretraining on Danish data;
25
- 2. `norallm/normistral-7b-warm`, continued pretraining on Norwegian data;
26
- 3. `timpal0l/Mistral-7B-v0.1-flashback-v2`, continued pretraining on Swedish data.
27
 
28
 
29
 
@@ -40,32 +39,22 @@ This is a DARE-TIES merge of the following models, all based on `mistralai/Mistr
40
  dict(
41
  model="danish-foundation-models/munin-7b-alpha",
42
  parameters=dict(
43
- density=0.9,
44
- weight=1.0,
45
- ),
46
- ),
47
- dict(
48
- model="norallm/normistral-7b-warm",
49
- parameters=dict(
50
- density=0.9,
51
  weight=1.0,
52
  ),
53
  ),
54
  dict(
55
  model="timpal0l/Mistral-7B-v0.1-flashback-v2",
56
  parameters=dict(
57
- density=0.9,
58
  weight=1.0,
59
  ),
60
  ),
61
  ],
62
- merge_method="dare_ties",
63
- random_seed=4242,
64
  base_model="mistralai/Mistral-7B-v0.1",
65
  parameters=dict(
66
- normalize=True,
67
  int8_mask=True,
 
68
  ),
69
- dtype="float16",
70
  )
71
  ```
 
19
 
20
  # ScandiMerge
21
 
22
+ This is a merge of the following models, all based on `mistralai/Mistral-7B-v0.1`:
23
 
24
  1. `danish-foundation-models/munin-7b-alpha`, continued pretraining on Danish data;
25
+ 2. `timpal0l/Mistral-7B-v0.1-flashback-v2`, continued pretraining on Swedish data.
 
26
 
27
 
28
 
 
39
  dict(
40
  model="danish-foundation-models/munin-7b-alpha",
41
  parameters=dict(
 
 
 
 
 
 
 
 
42
  weight=1.0,
43
  ),
44
  ),
45
  dict(
46
  model="timpal0l/Mistral-7B-v0.1-flashback-v2",
47
  parameters=dict(
 
48
  weight=1.0,
49
  ),
50
  ),
51
  ],
52
+ merge_method="task_arithmetic",
 
53
  base_model="mistralai/Mistral-7B-v0.1",
54
  parameters=dict(
 
55
  int8_mask=True,
56
+ normalize=True,
57
  ),
58
+ dtype="bfloat16",
59
  )
60
  ```