Hugging Face's logo

PygTesting
/
mystery-merge-nemo

Text Generation
Transformers
Safetensors
mistral
mergekit
Merge
conversational
text-generation-inference
Inference Endpoints
Model card Files Community

evo_model_test

This is a merge of pre-trained language models created using mergekit.

Merge Details

Merge Method

This model was merged using the DARE TIES merge method using ./evolve_merges/input_models/merge-10162024_972739363 as a base.

Models Merged

The following models were included in the merge:

  • ./evolve_merges/input_models/Magnum-Picaro-0.7-v2-12b_3809452655
  • ./evolve_merges/input_models/Chronos-Gold-12B-1.0_1861025797
  • ./evolve_merges/input_models/MN-12B-Mag-Mell-R1_399051020

Configuration

The following YAML configuration was used to produce this model:

base_model: ./evolve_merges/input_models/merge-10162024_972739363
dtype: bfloat16
merge_method: dare_ties
parameters:
  int8_mask: 1.0
  normalize: 1.0
slices:
- sources:
  - layer_range: [0, 4]
    model: ./evolve_merges/input_models/merge-10162024_972739363
    parameters:
      density:
      - filter: self_attn
        value: 0.6617851833521375
      - filter: mlp
        value: 1.0
      - value: 0.7758506135029611
      weight:
      - filter: self_attn
        value: 0.06553850894305135
      - filter: mlp
        value: 0.32372893196093133
      - value: 0.24761893893703177
  - layer_range: [0, 4]
    model: ./evolve_merges/input_models/Magnum-Picaro-0.7-v2-12b_3809452655
    parameters:
      density:
      - filter: self_attn
        value: 0.8619096186212604
      - filter: mlp
        value: 0.9632945037149085
      - value: 1.0
      weight:
      - filter: self_attn
        value: 0.5496368676404241
      - filter: mlp
        value: 0.2817627768141395
      - value: 0.2831242003449033
  - layer_range: [0, 4]
    model: ./evolve_merges/input_models/Chronos-Gold-12B-1.0_1861025797
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 1.0
      - value: 0.9238831652008582
      weight:
      - filter: self_attn
        value: 0.6983534009784523
      - filter: mlp
        value: 0.7786486269006042
      - value: 0.3362711484417948
  - layer_range: [0, 4]
    model: ./evolve_merges/input_models/MN-12B-Mag-Mell-R1_399051020
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 1.0
      - value: 0.897712174766424
      weight:
      - filter: self_attn
        value: 0.6494468053120542
      - filter: mlp
        value: 0.11769817501358182
      - value: 0.23745407940550356
- sources:
  - layer_range: [4, 8]
    model: ./evolve_merges/input_models/merge-10162024_972739363
    parameters:
      density:
      - filter: self_attn
        value: 0.768056839478356
      - filter: mlp
        value: 0.7392675781352855
      - value: 1.0
      weight:
      - filter: self_attn
        value: 0.4137398667324908
      - filter: mlp
        value: 0.5364761127195374
      - value: -0.06120952450996993
  - layer_range: [4, 8]
    model: ./evolve_merges/input_models/Magnum-Picaro-0.7-v2-12b_3809452655
    parameters:
      density:
      - filter: self_attn
        value: 0.9328263901133284
      - filter: mlp
        value: 1.0
      - value: 1.0
      weight:
      - filter: self_attn
        value: 0.512662918449004
      - filter: mlp
        value: 0.8133160093541117
      - value: 0.09518477923218693
  - layer_range: [4, 8]
    model: ./evolve_merges/input_models/Chronos-Gold-12B-1.0_1861025797
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 1.0
      - value: 1.0
      weight:
      - filter: self_attn
        value: 0.6534355737222919
      - filter: mlp
        value: -0.2733724467069448
      - value: 0.35896371241039604
  - layer_range: [4, 8]
    model: ./evolve_merges/input_models/MN-12B-Mag-Mell-R1_399051020
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 0.9645408518441749
      - value: 0.9920721804462888
      weight:
      - filter: self_attn
        value: 0.043888879112993606
      - filter: mlp
        value: 0.37533863309727755
      - value: 0.32692015564467836
- sources:
  - layer_range: [8, 12]
    model: ./evolve_merges/input_models/merge-10162024_972739363
    parameters:
      density:
      - filter: self_attn
        value: 0.9340306321054911
      - filter: mlp
        value: 1.0
      - value: 0.7968276665543247
      weight:
      - filter: self_attn
        value: 0.14846986084920036
      - filter: mlp
        value: 0.3955452929300913
      - value: 0.4270837195831495
  - layer_range: [8, 12]
    model: ./evolve_merges/input_models/Magnum-Picaro-0.7-v2-12b_3809452655
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 1.0
      - value: 1.0
      weight:
      - filter: self_attn
        value: 0.3649415030710907
      - filter: mlp
        value: 0.16275044387393922
      - value: 0.2758727640654811
  - layer_range: [8, 12]
    model: ./evolve_merges/input_models/Chronos-Gold-12B-1.0_1861025797
    parameters:
      density:
      - filter: self_attn
        value: 0.8295983370283204
      - filter: mlp
        value: 0.7788134370117827
      - value: 0.9398894811483364
      weight:
      - filter: self_attn
        value: 0.28746483121862637
      - filter: mlp
        value: 0.3358374043922244
      - value: 0.2275533582239845
  - layer_range: [8, 12]
    model: ./evolve_merges/input_models/MN-12B-Mag-Mell-R1_399051020
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 1.0
      - value: 0.727821766634972
      weight:
      - filter: self_attn
        value: 0.3081244623443608
      - filter: mlp
        value: 0.45014674558784984
      - value: 0.11047219740073362
- sources:
  - layer_range: [12, 16]
    model: ./evolve_merges/input_models/merge-10162024_972739363
    parameters:
      density:
      - filter: self_attn
        value: 0.6489316039694529
      - filter: mlp
        value: 1.0
      - value: 0.8272372022626591
      weight:
      - filter: self_attn
        value: 0.470708064142626
      - filter: mlp
        value: -0.047129110924588186
      - value: 0.42971949234723295
  - layer_range: [12, 16]
    model: ./evolve_merges/input_models/Magnum-Picaro-0.7-v2-12b_3809452655
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 0.6616234442454084
      - value: 1.0
      weight:
      - filter: self_attn
        value: 0.26282202905677127
      - filter: mlp
        value: 0.4448525732857457
      - value: 0.2229765978922556
  - layer_range: [12, 16]
    model: ./evolve_merges/input_models/Chronos-Gold-12B-1.0_1861025797
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 0.6135513085208061
      - value: 0.9581737790930396
      weight:
      - filter: self_attn
        value: 0.24444794214178578
      - filter: mlp
        value: 0.07937992720612315
      - value: -0.05228450555064985
  - layer_range: [12, 16]
    model: ./evolve_merges/input_models/MN-12B-Mag-Mell-R1_399051020
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 1.0
      - value: 1.0
      weight:
      - filter: self_attn
        value: 0.1719406804216106
      - filter: mlp
        value: 0.0934880168140769
      - value: 0.35045642161724166
- sources:
  - layer_range: [16, 20]
    model: ./evolve_merges/input_models/merge-10162024_972739363
    parameters:
      density:
      - filter: self_attn
        value: 0.5446785752563841
      - filter: mlp
        value: 0.8810586946591301
      - value: 0.9152297583356134
      weight:
      - filter: self_attn
        value: -0.0016341576761690624
      - filter: mlp
        value: -0.14493024949671152
      - value: 0.26832439639581773
  - layer_range: [16, 20]
    model: ./evolve_merges/input_models/Magnum-Picaro-0.7-v2-12b_3809452655
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 0.5944606032155147
      - value: 0.9302142529770252
      weight:
      - filter: self_attn
        value: 0.35950618403078893
      - filter: mlp
        value: 0.11051887834512175
      - value: 0.42291230769302385
  - layer_range: [16, 20]
    model: ./evolve_merges/input_models/Chronos-Gold-12B-1.0_1861025797
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 0.6546859569496538
      - value: 0.8503723026949942
      weight:
      - filter: self_attn
        value: 0.35331354069135923
      - filter: mlp
        value: 0.11666399796526544
      - value: 0.027977616826786067
  - layer_range: [16, 20]
    model: ./evolve_merges/input_models/MN-12B-Mag-Mell-R1_399051020
    parameters:
      density:
      - filter: self_attn
        value: 0.8237153213010172
      - filter: mlp
        value: 0.7779880619326531
      - value: 1.0
      weight:
      - filter: self_attn
        value: 0.7145318763470817
      - filter: mlp
        value: 0.4104048815986916
      - value: 0.07468194955613425
- sources:
  - layer_range: [20, 24]
    model: ./evolve_merges/input_models/merge-10162024_972739363
    parameters:
      density:
      - filter: self_attn
        value: 0.5231923060339636
      - filter: mlp
        value: 1.0
      - value: 0.9856713754180749
      weight:
      - filter: self_attn
        value: 0.4081014822719611
      - filter: mlp
        value: 0.09758488254406042
      - value: 0.3348194266336727
  - layer_range: [20, 24]
    model: ./evolve_merges/input_models/Magnum-Picaro-0.7-v2-12b_3809452655
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 1.0
      - value: 1.0
      weight:
      - filter: self_attn
        value: 0.7490383834336071
      - filter: mlp
        value: 0.4662047924812158
      - value: -0.24858277913931304
  - layer_range: [20, 24]
    model: ./evolve_merges/input_models/Chronos-Gold-12B-1.0_1861025797
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 1.0
      - value: 0.8502797089454639
      weight:
      - filter: self_attn
        value: 0.276884170342346
      - filter: mlp
        value: 0.633656940319029
      - value: 0.5235799339573071
  - layer_range: [20, 24]
    model: ./evolve_merges/input_models/MN-12B-Mag-Mell-R1_399051020
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 0.8562223977334964
      - value: 0.9716150483673114
      weight:
      - filter: self_attn
        value: 0.5270260765195226
      - filter: mlp
        value: 0.32711936701658684
      - value: 0.05670152518434478
- sources:
  - layer_range: [24, 28]
    model: ./evolve_merges/input_models/merge-10162024_972739363
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 1.0
      - value: 0.8553635955278736
      weight:
      - filter: self_attn
        value: 0.35406982791511876
      - filter: mlp
        value: -0.11643971781340703
      - value: 0.20075532527415488
  - layer_range: [24, 28]
    model: ./evolve_merges/input_models/Magnum-Picaro-0.7-v2-12b_3809452655
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 0.87297120460794
      - value: 1.0
      weight:
      - filter: self_attn
        value: 0.07480839031742999
      - filter: mlp
        value: 0.18311115096539785
      - value: 0.3625508152553395
  - layer_range: [24, 28]
    model: ./evolve_merges/input_models/Chronos-Gold-12B-1.0_1861025797
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 1.0
      - value: 1.0
      weight:
      - filter: self_attn
        value: 0.494667527482752
      - filter: mlp
        value: 0.3944202674139632
      - value: -0.19227439649461792
  - layer_range: [24, 28]
    model: ./evolve_merges/input_models/MN-12B-Mag-Mell-R1_399051020
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 1.0
      - value: 1.0
      weight:
      - filter: self_attn
        value: 0.06851638816347627
      - filter: mlp
        value: 0.431372227001768
      - value: 0.1747985843980182
- sources:
  - layer_range: [28, 32]
    model: ./evolve_merges/input_models/merge-10162024_972739363
    parameters:
      density:
      - filter: self_attn
        value: 0.9094528371038374
      - filter: mlp
        value: 1.0
      - value: 0.6090545725123906
      weight:
      - filter: self_attn
        value: 0.25309591486694805
      - filter: mlp
        value: -0.263292487608102
      - value: 0.1323202337738385
  - layer_range: [28, 32]
    model: ./evolve_merges/input_models/Magnum-Picaro-0.7-v2-12b_3809452655
    parameters:
      density:
      - filter: self_attn
        value: 0.6494843615875994
      - filter: mlp
        value: 1.0
      - value: 0.7515064103597758
      weight:
      - filter: self_attn
        value: 0.07729701084822604
      - filter: mlp
        value: 0.2170958326731126
      - value: 0.22214702687265422
  - layer_range: [28, 32]
    model: ./evolve_merges/input_models/Chronos-Gold-12B-1.0_1861025797
    parameters:
      density:
      - filter: self_attn
        value: 0.8431056158343985
      - filter: mlp
        value: 0.8838909258744341
      - value: 0.35295455870641634
      weight:
      - filter: self_attn
        value: 0.6551015978225493
      - filter: mlp
        value: 0.016410780482769546
      - value: 0.6370635339121399
  - layer_range: [28, 32]
    model: ./evolve_merges/input_models/MN-12B-Mag-Mell-R1_399051020
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 1.0
      - value: 1.0
      weight:
      - filter: self_attn
        value: 0.04318024669287196
      - filter: mlp
        value: 0.7642269685567962
      - value: 0.26850603466331324
- sources:
  - layer_range: [32, 36]
    model: ./evolve_merges/input_models/merge-10162024_972739363
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 1.0
      - value: 0.579520070097527
      weight:
      - filter: self_attn
        value: -0.051737601944818495
      - filter: mlp
        value: 0.3503787657405606
      - value: 0.08607827555366553
  - layer_range: [32, 36]
    model: ./evolve_merges/input_models/Magnum-Picaro-0.7-v2-12b_3809452655
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 1.0
      - value: 1.0
      weight:
      - filter: self_attn
        value: 0.28766985337224327
      - filter: mlp
        value: 0.3046959778412749
      - value: -0.0005520428411238121
  - layer_range: [32, 36]
    model: ./evolve_merges/input_models/Chronos-Gold-12B-1.0_1861025797
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 0.915429997855087
      - value: 1.0
      weight:
      - filter: self_attn
        value: 0.440410051026902
      - filter: mlp
        value: -0.21574554516791783
      - value: 0.15656972383477347
  - layer_range: [32, 36]
    model: ./evolve_merges/input_models/MN-12B-Mag-Mell-R1_399051020
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 1.0
      - value: 1.0
      weight:
      - filter: self_attn
        value: 0.3263876152481672
      - filter: mlp
        value: -0.040618303294953154
      - value: 0.47900376528192473
- sources:
  - layer_range: [36, 40]
    model: ./evolve_merges/input_models/merge-10162024_972739363
    parameters:
      density:
      - filter: self_attn
        value: 0.9171778237104341
      - filter: mlp
        value: 0.7229727777891508
      - value: 0.9122033861491662
      weight:
      - filter: self_attn
        value: 0.6154987734241069
      - filter: mlp
        value: 0.3910860949496661
      - value: 0.5286422728941228
  - layer_range: [36, 40]
    model: ./evolve_merges/input_models/Magnum-Picaro-0.7-v2-12b_3809452655
    parameters:
      density:
      - filter: self_attn
        value: 0.6023409600465159
      - filter: mlp
        value: 1.0
      - value: 1.0
      weight:
      - filter: self_attn
        value: 0.39644253937030505
      - filter: mlp
        value: 0.7570672338863116
      - value: 0.10261227723433294
  - layer_range: [36, 40]
    model: ./evolve_merges/input_models/Chronos-Gold-12B-1.0_1861025797
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 1.0
      - value: 0.8342554461687561
      weight:
      - filter: self_attn
        value: 0.4563403174251752
      - filter: mlp
        value: 0.313992481082509
      - value: 0.022583139471508834
  - layer_range: [36, 40]
    model: ./evolve_merges/input_models/MN-12B-Mag-Mell-R1_399051020
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 0.9211392650515542
      - value: 1.0
      weight:
      - filter: self_attn
        value: -0.17092104595693997
      - filter: mlp
        value: 0.13032109680489912
      - value: -0.03480332269062497
Downloads last month
211
Safetensors
Model size
12.2B params
Tensor type
BF16
·
Inference Providers NEW
Text Generation
This model is not currently available via any of the supported third-party Inference Providers, and the model is not deployed on the HF Inference API.

Model tree for PygTesting/mystery-merge-nemo

Finetunes
1 model