Hugging Face's logo

utter-project
/
EuroLLM-1.7B-Instruct

Safetensors
llama
Model card Files Community
3
phmartins commited on
Commit
f7789a3
•
1 Parent(s): aacb92a

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +14 -14
README.md CHANGED
@@ -97,17 +97,18 @@ Here is a summary of the model hyper-parameters:
97
  print(tokenizer.decode(outputs[0], skip_special_tokens=True))
98
 
99
 
 
100
  ## Results
101
 
102
  ### Machine Translation
103
 
104
- We evaluate EuroLLM-1.7B-Instruct on several machine translation benchmarks: FLORES-200, WMT-23, and WMT-24 comparing it with [Gemma-2B](https://huggingface.co/google/gemma-2b) and [Gemma-7B](https://huggingface.co/google/gemma-7b) (also instruction tuned on EuroBlocks-v0.1).
105
  The results show that EuroLLM-1.7B is substantially better than Gemma-2B in Machine Translation and competitive with Gemma-7B.
106
 
107
  #### Flores-200
108
  | Model | AVG | AVG en-xx | AVG xx-en | en-ar | en-bg | en-ca | en-cs | en-da | en-de | en-el | en-es-latam | en-et | en-fi | en-fr | en-ga | en-gl | en-hi | en-hr | en-hu | en-it | en-ja | en-ko | en-lt | en-lv | en-mt | en-nl | en-no | en-pl | en-pt-br | en-ro | en-ru | en-sk | en-sl | en-sv | en-tr | en-uk | en-zh-cn | ar-en | bg-en | ca-en | cs-en | da-en | de-en | el-en | es-latam-en | et-en | fi-en | fr-en | ga-en | gl-en | hi-en | hr-en | hu-en | it-en | ja-en | ko-en | lt-en | lv-en | mt-en | nl-en | no-en | pl-en | pt-br-en | ro-en | ru-en | sk-en | sl-en | sv-en | tr-en | uk-en | zh-cn-en |
109
  |--------------------------------|------|-----------|-----------|-------|-------|-------|-------|-------|-------|-------|--------------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|----------|-------|-------|-------|-------|-------|-------|-------|----------|-------|-------|-------|-------|-------|-------|-------|--------------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|----------|-------|-------|-------|-------|-------|-------|-------|----------|
110
- | EuroLLM-1.7B-Instruct | 86.75| 86.49| 87.01| 85.00| 89.37| 84.65| 89.23| 89.54| 87.00| 87.68| 86.43| 88.66| 89.18| 87.65| 74.66| 86.68| 76.94| 84.86| 86.43| 88.19| 89.45| 87.33| 87.73| 87.74| 67.66| 87.16| 90.08| 88.10| 89.39| 89.39| 88.11| 88.13| 87.08| 89.67| 87.21| 87.76| 86.45| 86.15| 87.46| 87.49| 87.97| 89.65| 88.80| 86.88| 86.70| 88.19| 88.66| 88.93| 81.95| 87.38| 87.97| 86.69| 87.15| 87.69| 86.78| 86.76| 85.39| 86.23| 76.93| 87.02| 90.24| 85.54| 89.22| 88.81| 86.02| 87.24| 86.81| 89.55| 87.76| 86.37| 86.00|
111
  | Gemma-2B-EuroBlocks | 81.56| 78.93 | 84.18 | 75.25 | 82.46 | 83.17 | 82.17 | 84.40 | 83.20 | 79.63 | 84.15 | 72.63 | 81.00 | 85.12 | 38.79 | 82.00 | 67.00 | 81.18 | 78.24 | 84.80 | 87.08 | 82.04 | 73.02 | 68.41 | 56.67 | 83.30 | 86.69 | 83.07 | 86.82 | 84.00 | 84.55 | 77.93 | 76.19 | 80.77 | 79.76 | 84.19 | 84.10 | 83.67 | 85.73 | 86.89 | 86.38 | 88.39 | 88.11 | 84.68 | 86.11 | 83.45 | 86.45 | 88.22 | 50.88 | 86.44 | 85.87 | 85.33 | 85.16 | 86.75 | 85.62 | 85.00 | 81.55 | 81.45 | 67.90 | 85.95 | 89.05 | 84.18 | 88.27 | 87.38 | 85.13 | 85.22 | 83.86 | 87.83 | 84.96 | 85.15 | 85.10 |
112
  | Gemma-7B-EuroBlocks | 86.16| 85.49 | 86.82 | 83.39 | 88.32 | 85.82 | 88.88 | 89.01 | 86.96 | 86.62 | 86.31 | 84.42 | 88.11 | 87.46 | 61.85 | 86.10 | 77.91 | 87.01 | 85.81 | 87.57 | 89.88 | 87.24 | 84.47 | 83.15 | 67.13 | 86.50 | 90.44 | 87.57 | 89.22 | 89.13 | 88.58 | 86.73 | 84.68 | 88.16 | 86.87 | 88.40 | 87.11 | 86.65 | 87.25 | 88.17 | 87.47 | 89.59 | 88.44 | 86.76 | 86.66 | 87.55 | 88.88 | 88.86 | 73.46 | 87.63 | 88.43 | 87.12 | 87.31 | 87.49 | 87.20 | 87.15 | 85.16 | 85.96 | 78.39 | 86.73 | 90.52 | 85.38 | 89.17 | 88.75 | 86.35 | 86.82 | 86.21 | 89.39 | 88.20 | 86.45 | 86.28 |
113
 
@@ -115,7 +116,7 @@ The results show that EuroLLM-1.7B is substantially better than Gemma-2B in Mach
115
  #### WMT-23
116
  | Model | AVG | AVG en-xx | AVG xx-en | AVG xx-xx | en-de | en-cs | en-uk | en-ru | en-zh-cn | de-en | uk-en | ru-en | zh-cn-en | cs-uk |
117
  |--------------------------------|------|-----------|-----------|-----------|-------|-------|-------|-------|----------|-------|-------|-------|----------|-------|
118
- | EuroLLM-1.7B-Instruct | 83.13 | 82.91 | 82.48 | 86.87| 81.33| 85.42| 81.61| 82.57| 83.62| 84.24| 85.36| 81.56| 78.76| 86.87 |
119
  | Gemma-2B-EuroBlocks | 79.86| 78.35 | 81.32 | 81.56 | 76.54 | 76.35 | 77.62 | 78.88 | 82.36 | 82.85 | 83.83 | 80.17 | 78.42 | 81.56 |
120
  | Gemma-7B-EuroBlocks | 83.90| 83.70 | 83.21 | 87.61 | 82.15 | 84.68 | 83.05 | 83.85 | 84.79 | 84.40 | 85.86 | 82.55 | 80.01 | 87.61 |
121
 
@@ -124,24 +125,23 @@ The results show that EuroLLM-1.7B is substantially better than Gemma-2B in Mach
124
  | Model | AVG | AVG en-xx | AVG xx-xx | en-es-latam | en-cs | en-ru | en-uk | en-ja | en-zh-cn | en-hi | cs-uk | ja-zh-cn |
125
  |---------|------|------|-------|-------|-------|-------|--------|--------|-------|-------|-------|-----|
126
  | EuroLLM-1.7B-Instruct|79.35|79.45|78.96|79.20|81.17|80.82|79.00|80.54|82.39|80.80|71.69|83.16|74.76|
127
- |Gemma-2B-EuroBlocks | 74.71|74.25|76.57|75.21|78.84|70.40|74.44|75.55|78.32|78.70|62.51|79.97|73.17|
128
- |Gemma-7B-EuroBlocks | 80.88|80.45|82.60|80.43|81.91|80.14|80.32|82.17|84.08|81.86|72.71|85.55|79.65|
129
 
130
  ### General Benchmarks
131
- We also compare EuroLLM-1.7B with [TinyLlama-1.1-3T](https://huggingface.co/TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T) and [Gemma-2B](https://huggingface.co/google/gemma-2b) on 3 general benchmarks: Arc Challenge and Hellaswag.
132
  For the non-english languages we use the [Okapi](https://aclanthology.org/2023.emnlp-demo.28.pdf) datasets.
133
- Results show that EuroLLM-1.7B is superior to TinyLlama-1.1-3T and similar to Gemma-2B on Hellaswag but worse on Arc Challenge and MMLU. This can be due to the lower number of parameters of EuroLLM-1.7B (1.133B non-embedding parameters against 1.981B).
134
 
135
  #### Arc Challenge
136
  | Model | Average | English | German | Spanish | French | Italian | Portuguese | Chinese | Russian | Dutch | Arabic | Swedish | Hindi | Hungarian | Romanian | Ukrainian | Danish | Catalan |
137
  |--------------------|---------|---------|--------|---------|--------|---------|------------|---------|---------|-------|--------|---------|--------|-----------|----------|-----------|--------|---------|
138
- | EuroLLM-1.7B | 0.3130 | 0.4215 | 0.3148 | 0.3376 | 0.3259 | 0.3396 | 0.3410 | 0.3068 | 0.2626 | 0.3037| 0.2652 | 0.3279 | 0.2688 | 0.3039 | 0.3085 | 0.2943 | 0.2956 | 0.3027 |
139
- | TinyLlama-1.1-3T | 0.2621 | 0.3473 | 0.2541 | 0.2726 | 0.2797 | 0.2643 | 0.2829 | 0.2573 | 0.2421 | 0.2404| 0.2335 | 0.2661 | 0.2337 | 0.244 | 0.2536 | 0.2626 | 0.2476 | 0.2736 |
140
- | Gemma-2B | 0.3617 | 0.4846 | 0.3755 | 0.3940 | 0.4080 | 0.3687 | 0.3872 | 0.3726 | 0.3456 | 0.3328| 0.3122 | 0.3519 | 0.2851 | 0.3039 | 0.3590 | 0.3601 | 0.3565 | 0.3516 |
141
-
142
  #### Hellaswag
143
  | Model | Average | English | German | Spanish | French | Italian | Portuguese | Russian | Dutch | Arabic | Swedish | Hindi | Hungarian | Romanian | Ukrainian | Danish | Catalan |
144
  |--------------------|---------|---------|--------|---------|--------|---------|------------|---------|--------|--------|---------|--------|-----------|----------|-----------|--------|---------|
145
- | EuroLLM-1.7B | 0.4653 | 0.6199 | 0.4653 | 0.5187 | 0.5173 | 0.5024 | 0.5116 | 0.4582 | 0.4821 | 0.3939 | 0.4722 | 0.3505 | 0.3970 | 0.4441 | 0.4224 | 0.4556 | 0.4329 |
146
- | TinyLlama-1.1-3T | 0.3710 | 0.6027 | 0.3652 | 0.4136 | 0.4104 | 0.3780 | 0.4008 | 0.3544 | 0.3637 | 0.2981 | 0.3569 | 0.2904 | 0.3147 | 0.3337 | 0.3440 | 0.3464 | 0.3628 |
147
- | Gemma-2B | 0.4666 | 0.7165 | 0.4756 | 0.5414 | 0.5180 | 0.4841 | 0.5081 | 0.4664 | 0.4655 | 0.3868 | 0.4383 | 0.3413 | 0.3710 | 0.4316 | 0.4291 | 0.4471 | 0.4448 |
 
97
  print(tokenizer.decode(outputs[0], skip_special_tokens=True))
98
 
99
 
100
+
101
  ## Results
102
 
103
  ### Machine Translation
104
 
105
+ We evaluate EuroLLM-1.7B-Instruct on several machine translation benchmarks: FLORES-200, WMT-23, and WMT-24 comparing it with [Gemma-2B](https://huggingface.co/google/gemma-2b) and [Gemma-7B](https://huggingface.co/google/gemma-7b) (also instruction tuned on EuroBlocks).
106
  The results show that EuroLLM-1.7B is substantially better than Gemma-2B in Machine Translation and competitive with Gemma-7B.
107
 
108
  #### Flores-200
109
  | Model | AVG | AVG en-xx | AVG xx-en | en-ar | en-bg | en-ca | en-cs | en-da | en-de | en-el | en-es-latam | en-et | en-fi | en-fr | en-ga | en-gl | en-hi | en-hr | en-hu | en-it | en-ja | en-ko | en-lt | en-lv | en-mt | en-nl | en-no | en-pl | en-pt-br | en-ro | en-ru | en-sk | en-sl | en-sv | en-tr | en-uk | en-zh-cn | ar-en | bg-en | ca-en | cs-en | da-en | de-en | el-en | es-latam-en | et-en | fi-en | fr-en | ga-en | gl-en | hi-en | hr-en | hu-en | it-en | ja-en | ko-en | lt-en | lv-en | mt-en | nl-en | no-en | pl-en | pt-br-en | ro-en | ru-en | sk-en | sl-en | sv-en | tr-en | uk-en | zh-cn-en |
110
  |--------------------------------|------|-----------|-----------|-------|-------|-------|-------|-------|-------|-------|--------------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|----------|-------|-------|-------|-------|-------|-------|-------|----------|-------|-------|-------|-------|-------|-------|-------|--------------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|----------|-------|-------|-------|-------|-------|-------|-------|----------|
111
+ | EuroLLM-1.7B-Instruct | 86.75| 86.49| 87.01|85.00| 89.37| 84.65| 89.23| 89.54| 87.00| 87.68| 86.43| 88.66| 89.18| 87.65| 74.66| 86.68| 76.94| 84.86| 86.43| 88.19| 89.45| 87.33| 87.73| 87.74| 67.66| 87.16| 90.08| 88.10| 89.39| 89.39| 88.11| 88.13| 87.08| 89.67| 87.21| 87.76| 86.45| 86.15| 87.46| 87.49| 87.97| 89.65| 88.80| 86.88| 86.70| 88.19| 88.66| 88.93| 81.95| 87.38| 87.97| 86.69| 87.15| 87.69| 86.78| 86.76| 85.39| 86.23| 76.93| 87.02| 90.24| 85.54| 89.22| 88.81| 86.02| 87.24| 86.81| 89.55| 87.76| 86.37| 86.00 |
112
  | Gemma-2B-EuroBlocks | 81.56| 78.93 | 84.18 | 75.25 | 82.46 | 83.17 | 82.17 | 84.40 | 83.20 | 79.63 | 84.15 | 72.63 | 81.00 | 85.12 | 38.79 | 82.00 | 67.00 | 81.18 | 78.24 | 84.80 | 87.08 | 82.04 | 73.02 | 68.41 | 56.67 | 83.30 | 86.69 | 83.07 | 86.82 | 84.00 | 84.55 | 77.93 | 76.19 | 80.77 | 79.76 | 84.19 | 84.10 | 83.67 | 85.73 | 86.89 | 86.38 | 88.39 | 88.11 | 84.68 | 86.11 | 83.45 | 86.45 | 88.22 | 50.88 | 86.44 | 85.87 | 85.33 | 85.16 | 86.75 | 85.62 | 85.00 | 81.55 | 81.45 | 67.90 | 85.95 | 89.05 | 84.18 | 88.27 | 87.38 | 85.13 | 85.22 | 83.86 | 87.83 | 84.96 | 85.15 | 85.10 |
113
  | Gemma-7B-EuroBlocks | 86.16| 85.49 | 86.82 | 83.39 | 88.32 | 85.82 | 88.88 | 89.01 | 86.96 | 86.62 | 86.31 | 84.42 | 88.11 | 87.46 | 61.85 | 86.10 | 77.91 | 87.01 | 85.81 | 87.57 | 89.88 | 87.24 | 84.47 | 83.15 | 67.13 | 86.50 | 90.44 | 87.57 | 89.22 | 89.13 | 88.58 | 86.73 | 84.68 | 88.16 | 86.87 | 88.40 | 87.11 | 86.65 | 87.25 | 88.17 | 87.47 | 89.59 | 88.44 | 86.76 | 86.66 | 87.55 | 88.88 | 88.86 | 73.46 | 87.63 | 88.43 | 87.12 | 87.31 | 87.49 | 87.20 | 87.15 | 85.16 | 85.96 | 78.39 | 86.73 | 90.52 | 85.38 | 89.17 | 88.75 | 86.35 | 86.82 | 86.21 | 89.39 | 88.20 | 86.45 | 86.28 |
114
 
 
116
  #### WMT-23
117
  | Model | AVG | AVG en-xx | AVG xx-en | AVG xx-xx | en-de | en-cs | en-uk | en-ru | en-zh-cn | de-en | uk-en | ru-en | zh-cn-en | cs-uk |
118
  |--------------------------------|------|-----------|-----------|-----------|-------|-------|-------|-------|----------|-------|-------|-------|----------|-------|
119
+ | EuroLLM-1.7B-Instruct | 83.13 | 82.91 | 82.48 | 86.87| 81.33| 85.42| 81.61| 82.57| 83.62| 84.24| 85.36| 81.56| 78.76| 86.87 |
120
  | Gemma-2B-EuroBlocks | 79.86| 78.35 | 81.32 | 81.56 | 76.54 | 76.35 | 77.62 | 78.88 | 82.36 | 82.85 | 83.83 | 80.17 | 78.42 | 81.56 |
121
  | Gemma-7B-EuroBlocks | 83.90| 83.70 | 83.21 | 87.61 | 82.15 | 84.68 | 83.05 | 83.85 | 84.79 | 84.40 | 85.86 | 82.55 | 80.01 | 87.61 |
122
 
 
125
  | Model | AVG | AVG en-xx | AVG xx-xx | en-es-latam | en-cs | en-ru | en-uk | en-ja | en-zh-cn | en-hi | cs-uk | ja-zh-cn |
126
  |---------|------|------|-------|-------|-------|-------|--------|--------|-------|-------|-------|-----|
127
  | EuroLLM-1.7B-Instruct|79.35|79.45|78.96|79.20|81.17|80.82|79.00|80.54|82.39|80.80|71.69|83.16|74.76|
128
+ |Gemma-2B-EuroBlocks| 74.71|74.25|76.57|75.21|78.84|70.40|74.44|75.55|78.32|78.70|62.51|79.97|73.17|
129
+ |Gemma-7B-EuroBlocks| 80.88|80.45|82.60|80.43|81.91|80.14|80.32|82.17|84.08|81.86|72.71|85.55|79.65|
130
 
131
  ### General Benchmarks
132
+ We also compare EuroLLM-1.7B with [TinyLlama-v1.1](https://huggingface.co/TinyLlama/TinyLlama_v1.1) and [Gemma-2B](https://huggingface.co/google/gemma-2b) on 3 general benchmarks: Arc Challenge and Hellaswag.
133
  For the non-english languages we use the [Okapi](https://aclanthology.org/2023.emnlp-demo.28.pdf) datasets.
134
+ Results show that EuroLLM-1.7B is superior to TinyLlama-1.1-3T and similar to Gemma-2B on Hellaswag but worse on Arc Challenge. This can be due to the lower number of parameters of EuroLLM-1.7B (1.133B non-embedding parameters against 1.981B).
135
 
136
  #### Arc Challenge
137
  | Model | Average | English | German | Spanish | French | Italian | Portuguese | Chinese | Russian | Dutch | Arabic | Swedish | Hindi | Hungarian | Romanian | Ukrainian | Danish | Catalan |
138
  |--------------------|---------|---------|--------|---------|--------|---------|------------|---------|---------|-------|--------|---------|--------|-----------|----------|-----------|--------|---------|
139
+ | EuroLLM-1.7B-Instruct | 0.3268 | 0.3218 | 0.4070 | 0.3293 | 0.3521 | 0.3370 | 0.3422 | 0.3496 | 0.3060 | 0.3122 | 0.3174 | 0.2866 | 0.3373 | 0.2817 | 0.3031 | 0.3179 | 0.3199 | 0.3248 | 0.3310 |
140
+ | TinyLlama-v1.1 | 0.2650 | 0.2583 | 0.3712 | 0.2524 | 0.2795 | 0.2883 | 0.2652 | 0.2906 | 0.2410 | 0.2669 | 0.2404 | 0.2310 | 0.2687 | 0.2354 | 0.2449 | 0.2476 | 0.2524 | 0.2494 | 0.2796 |
141
+ | Gemma-2B | 0.3617 | 0.3540 | 0.4846 | 0.3755 | 0.3940 | 0.4080 | 0.3687 | 0.3872 | 0.3726 | 0.3456 | 0.3328 | 0.3122 | 0.3519 | 0.2851 | 0.3039 | 0.3590 | 0.3601 | 0.3565 | 0.3516 |
 
142
  #### Hellaswag
143
  | Model | Average | English | German | Spanish | French | Italian | Portuguese | Russian | Dutch | Arabic | Swedish | Hindi | Hungarian | Romanian | Ukrainian | Danish | Catalan |
144
  |--------------------|---------|---------|--------|---------|--------|---------|------------|---------|--------|--------|---------|--------|-----------|----------|-----------|--------|---------|
145
+ | EuroLLM-1.7B-Instruct | 0.4744 | 0.4654 | 0.6084 | 0.4772 | 0.5310 | 0.5260 | 0.5067 | 0.5206 | 0.4674 | 0.4893 | 0.4075 | 0.4813 | 0.3605 | 0.4067 | 0.4598 | 0.4368 | 0.4700 | 0.4405 |
146
+ | TinyLlama-v1.1 |0.3674 | 0.3503 | 0.6248 | 0.3650 | 0.4137 | 0.4010 | 0.3780 | 0.3892 | 0.3494 | 0.3588 | 0.2880 | 0.3561 | 0.2841 | 0.3073 | 0.3267 | 0.3349 | 0.3408 | 0.3613 |
147
+ | Gemma-2B |0.4666 | 0.4499 | 0.7165 | 0.4756 | 0.5414 | 0.5180 | 0.4841 | 0.5081 | 0.4664 | 0.4655 | 0.3868 | 0.4383 | 0.3413 | 0.3710 | 0.4316 | 0.4291 | 0.4471 | 0.4448 |