mapama247 commited on
Commit
4f5c9f1
1 Parent(s): 91108cd

add remaining sections in readme

Browse files
Files changed (1) hide show
  1. README.md +101 -1
README.md CHANGED
@@ -129,4 +129,104 @@ The accelerated partition is composed of 1,120 nodes with the following specific
129
  |7B|128|512|
130
  |40B|256 / 512|1,024 / 2,048|
131
 
132
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
129
  |7B|128|512|
130
  |40B|256 / 512|1,024 / 2,048|
131
 
132
+ ---
133
+
134
+ ## How to use
135
+
136
+ <span style="color:red">TODO: table 2B model</span>
137
+
138
+ ---
139
+
140
+ ## Data
141
+
142
+ <span style="color:red">TODO: table 2B model</span>
143
+
144
+ ---
145
+
146
+ ## Evaluation
147
+
148
+ <span style="color:red">TODO: table 2B model</span>
149
+
150
+ ## Ethical Considerations and Limitations
151
+
152
+ We examine the presence of undesired societal and cognitive biases present in this model using different benchmarks. For societal biases,
153
+ we test performance using the BBQ dataset (Parrish et al., 2022) in the original English and the Regard dataset (Sheng et al., 2019).
154
+ We report that while performance is high (accuracies between 0.69 and 0.87 depending on the social category) in disambiguated settings
155
+ the model performs very poorly in ambiguous settings, which is indicative of the presence of societal biases which need to be addressed in post-training phases.
156
+
157
+ We additionally analyse model generations using the Regard dataset and classifier in Catalan, Spanish, and English using backtranslation and manual revision of the
158
+ translations. We find no statistically significant difference in regard between majority and minority groups for any regard types,
159
+ with the exception of negative regard in Catalan where model generations are actually slightly worse for social majorities.
160
+ Our analyses on societal biases show that while these biases are capable of interfering with model performance as expressed in the results on the BBQ dataset,
161
+ their tendency for representational harm is limited given the results of the Regard dataset. We highlight that our analyses of these biases are by no means exhaustive
162
+ and are limited by the relative scarcity of adequate resources in all languages present in the training data. We aim to gradually extend and expand our analyses
163
+ in future work.
164
+
165
+ Our cognitive bias analysis focuses on positional effects in 0-shot settings, and majority class bias in few-shot settings.
166
+ For positional effects, we leverage the ARC Multiple Choice Question dataset (Clark et al., 2018).
167
+ We observe moderate to strong primacy effects, whereby the model shows a preference for answers towards the beginning of the list of provided answers.
168
+ We measure effects of majority class effects in few-shot settings using SST-2 (Socher et al., 2013). We detect moderate effects,
169
+ implying that outputs can be influenced by the prompts.
170
+
171
+ We highlight that these results can be expected from a pretrained model that has not yet been instruction-tuned or aligned.
172
+ These tests are performed in order to show the biases the model may contain.
173
+ We urge developers to take them into account and perform safety testing and tuning tailored to their specific applications of the model.
174
+
175
+ ---
176
+
177
+ ## Additional information
178
+
179
+ ### Author
180
+ The Language Technologies Unit from Barcelona Supercomputing Center.
181
+
182
+ ### Contact
183
+ For further information, please send an email to <langtech@bsc.es>.
184
+
185
+ ### Copyright
186
+ Copyright(c) 2024 by Language Technologies Unit, Barcelona Supercomputing Center.
187
+
188
+ ### Funding
189
+ This work has been promoted and financed by the Government of Catalonia through the [Aina Project](https://projecteaina.cat/).
190
+
191
+ This work is funded by the _Ministerio para la Transformación Digital y de la Función Pública_ - Funded by EU – NextGenerationEU
192
+ within the framework of [ILENIA Project](https://proyectoilenia.es/) with reference 2022/TL22/00215337, 2022/TL22/00215336, 2022/TL22/00215335, 2022/TL22/00215334.
193
+
194
+ ### Acknowledgements
195
+
196
+ This project benefited from the contributions of many teams and institutions, including:
197
+ Senado de España, Parlament de Catalunya, Òmnium Cultural, Dialnet, Institut d’Estudis Aranesos,
198
+ Fundación Elcano, Universidad de Las Palmas de Gran Canaria, Occiglot, Common Crawl, the Welsh Government,
199
+ the German Research Center for Artificial Intelligence (DFKI) and the partners of Proyecto ILENIA.
200
+ Their valuable efforts have been instrumental in the development of this work.
201
+
202
+ A special acknowledgment is reserved for the NVIDIA Team with whom we have been meeting on a regular basis.
203
+ Their consistent support has been particularly appreciated throughout the process.
204
+
205
+ ### Disclaimer
206
+ Be aware that the model may contain biases or other unintended distortions.
207
+ When third parties deploy systems or provide services based on this model, or use the model themselves,
208
+ they bear the responsibility for mitigating any associated risks and ensuring compliance with applicable regulations,
209
+ including those governing the use of Artificial Intelligence.
210
+
211
+ The Barcelona Supercomputing Center, as the owner and creator of the model, shall not be held liable for any outcomes resulting from third-party use.
212
+
213
+ ## Citation
214
+ <span style="color:red">Work in progress, paper coming soon.</span>
215
+ ```bibtext
216
+ @article{salamandra,
217
+ title={Salamandra Technical Report},
218
+ author={LangTech@BSC},
219
+ year={2024},
220
+ url = {}
221
+ }
222
+ ```
223
+
224
+ ## License
225
+ [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0)
226
+
227
+ ## Model Index
228
+ |Model|Base|Instruct|
229
+ |:---:|:---:|:---:|
230
+ |2B| WiP | WiP |
231
+ |7B| [Link](https://huggingface.co/projecte-aina/salamandra-7b) | [Link](https://huggingface.co/projecte-aina/salamandra-7b-instruct) |
232
+ |40B| WiP | WiP |