fblgit commited on
Commit
4b62fcc
•
1 Parent(s): eb838c2

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +23 -11
README.md CHANGED
@@ -111,12 +111,23 @@ model-index:
111
  source:
112
  url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=fblgit/UNA-SimpleSmaug-34b-v1beta
113
  name: Open LLM Leaderboard
 
 
114
  ---
115
 
116
  # UNA-SimpleSmaug-34b-v1beta
117
 
118
  Scoring 04-February-2024 #1 34B model, outperforming its original base model Smaug-34B-v0.1 with `77.41` 😎
119
- Oh, btw.. this one went thru SFT so the abacus inside Smaug is back to normal.. so you can further train/dpo him .. RESET!
 
 
 
 
 
 
 
 
 
120
 
121
  ![UNA](https://huggingface.co/fblgit/UNA-SimpleSmaug-34b-v1beta/resolve/main/unasimple.png)
122
  Applied UNA only on the Attention, not on the MLP's
@@ -132,7 +143,17 @@ Results: Improving mathematican and reasoning capabilities without degrading and
132
  **And enjoy our ModelSimilarities tool detector** https://github.com/fblgit/model-similarity where we confirmed numerically the bloodties of the model.
133
  ## Evals
134
 
135
- Pending, but so far this one
 
 
 
 
 
 
 
 
 
 
136
  ```
137
  | Task |Version| Metric |Value |
138
  |-------------|------:|--------|----------------:|
@@ -155,13 +176,4 @@ To abacusai for making Smaug-34B, the Bagel, and all the magic behind the base m
155
  # [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
156
  Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_fblgit__UNA-SimpleSmaug-34b-v1beta)
157
 
158
- | Metric |Value|
159
- |---------------------------------|----:|
160
- |Avg. |77.41|
161
- |AI2 Reasoning Challenge (25-Shot)|74.57|
162
- |HellaSwag (10-Shot) |86.74|
163
- |MMLU (5-Shot) |76.68|
164
- |TruthfulQA (0-shot) |70.17|
165
- |Winogrande (5-shot) |83.82|
166
- |GSM8k (5-shot) |72.48|
167
 
 
111
  source:
112
  url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=fblgit/UNA-SimpleSmaug-34b-v1beta
113
  name: Open LLM Leaderboard
114
+
115
+
116
  ---
117
 
118
  # UNA-SimpleSmaug-34b-v1beta
119
 
120
  Scoring 04-February-2024 #1 34B model, outperforming its original base model Smaug-34B-v0.1 with `77.41` 😎
121
+ Oh, btw.. this one went thru SFT so the abacus inside Smaug is back to normal.. so you can further train/dpo him .. RESET!..
122
+
123
+ *UPDATES* March : Stills undisputed 34B King
124
+ Smaug 70B stills undisputed 70B King
125
+
126
+ ====
127
+ And people wonders.. why there is no UNA of Hermes or Smaug 70B? << i dont think is worth the time to spend on a model that is widely known for not being too useful, likely UNA can fix some of the internal mess..
128
+ for Hermes, we spoke chitchat quick a couple times but nothing solid, but we would like to make a reborn of excellent models using UNA, just liek we did with UNA-Dolphin where we saw
129
+ relevant performance is short time.
130
+ ===
131
 
132
  ![UNA](https://huggingface.co/fblgit/UNA-SimpleSmaug-34b-v1beta/resolve/main/unasimple.png)
133
  Applied UNA only on the Attention, not on the MLP's
 
143
  **And enjoy our ModelSimilarities tool detector** https://github.com/fblgit/model-similarity where we confirmed numerically the bloodties of the model.
144
  ## Evals
145
 
146
+
147
+ | Metric |Value|
148
+ |---------------------------------|----:|
149
+ |Avg. |77.41|
150
+ |AI2 Reasoning Challenge (25-Shot)|74.57|
151
+ |HellaSwag (10-Shot) |86.74|
152
+ |MMLU (5-Shot) |76.68|
153
+ |TruthfulQA (0-shot) |70.17|
154
+ |Winogrande (5-shot) |83.82|
155
+ |GSM8k (5-shot) |72.48|
156
+
157
  ```
158
  | Task |Version| Metric |Value |
159
  |-------------|------:|--------|----------------:|
 
176
  # [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
177
  Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_fblgit__UNA-SimpleSmaug-34b-v1beta)
178
 
 
 
 
 
 
 
 
 
 
179