README.md

metadata

base_model:
  - lucyknada/microsoft_WizardLM-2-7B
  - HuggingFaceH4/zephyr-7b-beta
library_name: transformers
tags:
  - mergekit
  - merge
license: apache-2.0
pipeline_tag: text-generation

zephyr-beta-wizardLM-2-merge-7B

This is a merge of two pre-trained language models created using mergekit. The goal was to explore the impact of merging on reasoning and narrative generation. Given that both fine-tuned models are BF16 precision (despite the precision of the base model Mistral 7B v0.1 being FP16), this avoids issues with mixed precision during merging.

Merge Details

Merge Method

This model was merged using the SLERP merge method.

Models Merged

The following models were included in the merge:

• lucyknada/microsoft_WizardLM-2-7B
• HuggingFaceH4/zephyr-7b-beta

Configuration

The following YAML configuration was used to produce this model:

slices:
  - sources:
    - model: HuggingFaceH4/zephyr-7b-beta
      layer_range: [0,32]
    - model: lucyknada/microsoft_WizardLM-2-7B
      layer_range: [0,32]
merge_method: slerp
base_model: HuggingFaceH4/zephyr-7b-beta
parameters:
  t:
    - value: 0.5
dtype: bfloat16