Hugging Face's logo

Noodlz
/
WizardLaker-7B

Text Generation
Transformers
Safetensors
mistral
mergekit
Merge
text-generation-inference
Inference Endpoints
Model card Files Community
WizardLaker-7B / README.md
Noodlz's picture
Noodlz
Update README.md
1e0c5f5 verified
preview code
|
raw
history blame
1.44 kB
metadata
base_model: []
library_name: transformers
tags:
  - mergekit
  - merge
license: apache-2.0

WizardLaker 7B

image/png

This is a merge of the new WizardLM 2 7B model with my custom DolphinLake Model(https://huggingface.co/Noodlz/DolphinLake-7B). Seems to perform well. will be submitting for evals on openLLM leaderboards. Created using mergekit.

Merge Details

Merge Method

This model was merged using the DARE TIES merge method using amazingvince/Not-WizardLM-2-7B as a base.

Models Merged

The following models were included in the merge:

  • /Noodlz/DolphinLake-7B

Configuration

The following YAML configuration was used to produce this model:

merge_method: dare_ties

parameters:
  int8_mask: true
  t:
    - filter: self_attn
      value: [0, 0.5, 0.3, 0.7, 1]
    - filter: mlp
      value: [1, 0.5, 0.7, 0.3, 0]
    - value: 0.5 # fallback for rest of tensors
  embed_slerp: true

models:
  - model: amazingvince/Not-WizardLM-2-7B
    # No parameters necessary for base model
  - model: /Noodlz/DolphinLake-7B
    parameters:
      density: 0.58
      weight: 0.4

base_model: amazingvince/Not-WizardLM-2-7B

tokenizer_source: model:amazingvince/Not-WizardLM-2-7B
dtype: bfloat16