---
tags:
- llama
- merge
---
Five different instruction-tuned models (which I'm sure are intuitively obvious from the name) merged using the methodology described in [Resolving Interference When Merging Models](https://arxiv.org/abs/2306.01708).

In theory this should retain more of the capabilites of the constituent models than a straight linear merge would. In my testing, it feels quite capable.

Base model used for the merge: [TheBloke/Llama-2-13B-fp16](https://huggingface.co/TheBloke/Llama-2-13B-fp16)

Models merged in:
* [OpenOrca-Platypus2-13B](https://huggingface.co/Open-Orca/OpenOrca-Platypus2-13B)
* [limarp-13b-merged](https://huggingface.co/Oniichat/limarp-13b-merged)
* [Nous-Hermes-Llama2-13b](https://huggingface.co/NousResearch/Nous-Hermes-Llama2-13b)
* [chronos-13b-v2](https://huggingface.co/elinas/chronos-13b-v2)
* [airoboros-l2-13b-gpt4-1.4.1](https://huggingface.co/jondurbin/airoboros-l2-13b-gpt4-1.4.1)

Works quite well with Alpaca-style prompts:
```
### Instruction:

...

### Response:
```

The script I used to perform the merge is available [here](https://github.com/cg123/ties-merge).

The command that produced this model:
```
python ties_merge.py TheBloke/Llama-2-13B-fp16 ./Chronorctypus-Limarobormes-13b --merge elinas/chronos-13b-v2 --merge Open-Orca/OpenOrca-Platypus2-13B --merge Oniichat/limarp-13b-merged --merge jondurbin/airoboros-l2-13b-gpt4-1.4.1 --merge NousResearch/Nous-Hermes-Llama2-13b --cuda
```
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_chargoddard__Chronorctypus-Limarobormes-13b)

| Metric                | Value                     |
|-----------------------|---------------------------|
| Avg.                  | 49.88   |
| ARC (25-shot)         | 59.9          |
| HellaSwag (10-shot)   | 82.75    |
| MMLU (5-shot)         | 58.45         |
| TruthfulQA (0-shot)   | 51.9   |
| Winogrande (5-shot)   | 74.43   |
| GSM8K (5-shot)        | 3.87        |
| DROP (3-shot)         | 17.89         |