Update README.md
Browse files
README.md
CHANGED
@@ -97,10 +97,21 @@ model-index:
|
|
97 |
name: Open LLM Leaderboard
|
98 |
---
|
99 |
|
100 |
-
This is a new kind of model optimization.
|
101 |
-
This model is based on MaziyarPanahi/calme-2.1-qwen2-72b, which was tuned from Qwen2-72B.
|
102 |
|
103 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
104 |
|
105 |
This research was supported with hardware from the [appliedAI Institute](https://www.appliedai-institute.de/en/), whose goal is to generate and communicate high-quality knowledge about trustworthy AI.
|
106 |
|
|
|
97 |
name: Open LLM Leaderboard
|
98 |
---
|
99 |
|
100 |
+
This is a new kind of model optimization. It is based on a new method for the analysis of the functional role of layers within the transformer stack, and on layer duplication (self-merging) to increase intelligence.
|
|
|
101 |
|
102 |
+
*No Weights were modified in this process!*
|
103 |
+
|
104 |
+
### Model improvement (%) with layer duplication:
|
105 |
+
| | Average | IFEval | BBH | MATH Lvl 5 | GPQA | MUSR | MMLU-PRO |
|
106 |
+
|-----------------|---------|--------|------|------------|------|-------|----------|
|
107 |
+
| RYS Improvement | 2.61 | -2.05 | 2.51 | 8.16 | 2.58 | 17.72 | 0.31 |
|
108 |
+
|
109 |
+
|
110 |
+
This model is based on MaziyarPanahi/calme-2.1-qwen2-72b, which was tuned from Qwen2-72B. As this method is orthogonal to fine-tuning, the further finetune from MaziyarPanahi now has the top position:
|
111 |
+
https://huggingface.co/MaziyarPanahi/calme-2.4-rys-78b
|
112 |
+
|
113 |
+
|
114 |
+
A paper on the technique is currently being written. Currently, all four top models on the leaderboard are based on the RYS method. Special thanks to my wife, for putting up with me coding in the basement for too many evenings and weekends for months!
|
115 |
|
116 |
This research was supported with hardware from the [appliedAI Institute](https://www.appliedai-institute.de/en/), whose goal is to generate and communicate high-quality knowledge about trustworthy AI.
|
117 |
|