Update README.md
Browse files
README.md
CHANGED
@@ -111,12 +111,23 @@ model-index:
|
|
111 |
source:
|
112 |
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=fblgit/UNA-SimpleSmaug-34b-v1beta
|
113 |
name: Open LLM Leaderboard
|
|
|
|
|
114 |
---
|
115 |
|
116 |
# UNA-SimpleSmaug-34b-v1beta
|
117 |
|
118 |
Scoring 04-February-2024 #1 34B model, outperforming its original base model Smaug-34B-v0.1 with `77.41` 😎
|
119 |
-
Oh, btw.. this one went thru SFT so the abacus inside Smaug is back to normal.. so you can further train/dpo him .. RESET
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
120 |
|
121 |
![UNA](https://huggingface.co/fblgit/UNA-SimpleSmaug-34b-v1beta/resolve/main/unasimple.png)
|
122 |
Applied UNA only on the Attention, not on the MLP's
|
@@ -132,7 +143,17 @@ Results: Improving mathematican and reasoning capabilities without degrading and
|
|
132 |
**And enjoy our ModelSimilarities tool detector** https://github.com/fblgit/model-similarity where we confirmed numerically the bloodties of the model.
|
133 |
## Evals
|
134 |
|
135 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
136 |
```
|
137 |
| Task |Version| Metric |Value |
|
138 |
|-------------|------:|--------|----------------:|
|
@@ -155,13 +176,4 @@ To abacusai for making Smaug-34B, the Bagel, and all the magic behind the base m
|
|
155 |
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
|
156 |
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_fblgit__UNA-SimpleSmaug-34b-v1beta)
|
157 |
|
158 |
-
| Metric |Value|
|
159 |
-
|---------------------------------|----:|
|
160 |
-
|Avg. |77.41|
|
161 |
-
|AI2 Reasoning Challenge (25-Shot)|74.57|
|
162 |
-
|HellaSwag (10-Shot) |86.74|
|
163 |
-
|MMLU (5-Shot) |76.68|
|
164 |
-
|TruthfulQA (0-shot) |70.17|
|
165 |
-
|Winogrande (5-shot) |83.82|
|
166 |
-
|GSM8k (5-shot) |72.48|
|
167 |
|
|
|
111 |
source:
|
112 |
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=fblgit/UNA-SimpleSmaug-34b-v1beta
|
113 |
name: Open LLM Leaderboard
|
114 |
+
|
115 |
+
|
116 |
---
|
117 |
|
118 |
# UNA-SimpleSmaug-34b-v1beta
|
119 |
|
120 |
Scoring 04-February-2024 #1 34B model, outperforming its original base model Smaug-34B-v0.1 with `77.41` 😎
|
121 |
+
Oh, btw.. this one went thru SFT so the abacus inside Smaug is back to normal.. so you can further train/dpo him .. RESET!..
|
122 |
+
|
123 |
+
*UPDATES* March : Stills undisputed 34B King
|
124 |
+
Smaug 70B stills undisputed 70B King
|
125 |
+
|
126 |
+
====
|
127 |
+
And people wonders.. why there is no UNA of Hermes or Smaug 70B? << i dont think is worth the time to spend on a model that is widely known for not being too useful, likely UNA can fix some of the internal mess..
|
128 |
+
for Hermes, we spoke chitchat quick a couple times but nothing solid, but we would like to make a reborn of excellent models using UNA, just liek we did with UNA-Dolphin where we saw
|
129 |
+
relevant performance is short time.
|
130 |
+
===
|
131 |
|
132 |
![UNA](https://huggingface.co/fblgit/UNA-SimpleSmaug-34b-v1beta/resolve/main/unasimple.png)
|
133 |
Applied UNA only on the Attention, not on the MLP's
|
|
|
143 |
**And enjoy our ModelSimilarities tool detector** https://github.com/fblgit/model-similarity where we confirmed numerically the bloodties of the model.
|
144 |
## Evals
|
145 |
|
146 |
+
|
147 |
+
| Metric |Value|
|
148 |
+
|---------------------------------|----:|
|
149 |
+
|Avg. |77.41|
|
150 |
+
|AI2 Reasoning Challenge (25-Shot)|74.57|
|
151 |
+
|HellaSwag (10-Shot) |86.74|
|
152 |
+
|MMLU (5-Shot) |76.68|
|
153 |
+
|TruthfulQA (0-shot) |70.17|
|
154 |
+
|Winogrande (5-shot) |83.82|
|
155 |
+
|GSM8k (5-shot) |72.48|
|
156 |
+
|
157 |
```
|
158 |
| Task |Version| Metric |Value |
|
159 |
|-------------|------:|--------|----------------:|
|
|
|
176 |
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
|
177 |
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_fblgit__UNA-SimpleSmaug-34b-v1beta)
|
178 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
179 |
|