add remaining sections in readme
Browse files
README.md
CHANGED
@@ -129,4 +129,104 @@ The accelerated partition is composed of 1,120 nodes with the following specific
|
|
129 |
|7B|128|512|
|
130 |
|40B|256 / 512|1,024 / 2,048|
|
131 |
|
132 |
-
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
129 |
|7B|128|512|
|
130 |
|40B|256 / 512|1,024 / 2,048|
|
131 |
|
132 |
+
---
|
133 |
+
|
134 |
+
## How to use
|
135 |
+
|
136 |
+
<span style="color:red">TODO: table 2B model</span>
|
137 |
+
|
138 |
+
---
|
139 |
+
|
140 |
+
## Data
|
141 |
+
|
142 |
+
<span style="color:red">TODO: table 2B model</span>
|
143 |
+
|
144 |
+
---
|
145 |
+
|
146 |
+
## Evaluation
|
147 |
+
|
148 |
+
<span style="color:red">TODO: table 2B model</span>
|
149 |
+
|
150 |
+
## Ethical Considerations and Limitations
|
151 |
+
|
152 |
+
We examine the presence of undesired societal and cognitive biases present in this model using different benchmarks. For societal biases,
|
153 |
+
we test performance using the BBQ dataset (Parrish et al., 2022) in the original English and the Regard dataset (Sheng et al., 2019).
|
154 |
+
We report that while performance is high (accuracies between 0.69 and 0.87 depending on the social category) in disambiguated settings
|
155 |
+
the model performs very poorly in ambiguous settings, which is indicative of the presence of societal biases which need to be addressed in post-training phases.
|
156 |
+
|
157 |
+
We additionally analyse model generations using the Regard dataset and classifier in Catalan, Spanish, and English using backtranslation and manual revision of the
|
158 |
+
translations. We find no statistically significant difference in regard between majority and minority groups for any regard types,
|
159 |
+
with the exception of negative regard in Catalan where model generations are actually slightly worse for social majorities.
|
160 |
+
Our analyses on societal biases show that while these biases are capable of interfering with model performance as expressed in the results on the BBQ dataset,
|
161 |
+
their tendency for representational harm is limited given the results of the Regard dataset. We highlight that our analyses of these biases are by no means exhaustive
|
162 |
+
and are limited by the relative scarcity of adequate resources in all languages present in the training data. We aim to gradually extend and expand our analyses
|
163 |
+
in future work.
|
164 |
+
|
165 |
+
Our cognitive bias analysis focuses on positional effects in 0-shot settings, and majority class bias in few-shot settings.
|
166 |
+
For positional effects, we leverage the ARC Multiple Choice Question dataset (Clark et al., 2018).
|
167 |
+
We observe moderate to strong primacy effects, whereby the model shows a preference for answers towards the beginning of the list of provided answers.
|
168 |
+
We measure effects of majority class effects in few-shot settings using SST-2 (Socher et al., 2013). We detect moderate effects,
|
169 |
+
implying that outputs can be influenced by the prompts.
|
170 |
+
|
171 |
+
We highlight that these results can be expected from a pretrained model that has not yet been instruction-tuned or aligned.
|
172 |
+
These tests are performed in order to show the biases the model may contain.
|
173 |
+
We urge developers to take them into account and perform safety testing and tuning tailored to their specific applications of the model.
|
174 |
+
|
175 |
+
---
|
176 |
+
|
177 |
+
## Additional information
|
178 |
+
|
179 |
+
### Author
|
180 |
+
The Language Technologies Unit from Barcelona Supercomputing Center.
|
181 |
+
|
182 |
+
### Contact
|
183 |
+
For further information, please send an email to <langtech@bsc.es>.
|
184 |
+
|
185 |
+
### Copyright
|
186 |
+
Copyright(c) 2024 by Language Technologies Unit, Barcelona Supercomputing Center.
|
187 |
+
|
188 |
+
### Funding
|
189 |
+
This work has been promoted and financed by the Government of Catalonia through the [Aina Project](https://projecteaina.cat/).
|
190 |
+
|
191 |
+
This work is funded by the _Ministerio para la Transformación Digital y de la Función Pública_ - Funded by EU – NextGenerationEU
|
192 |
+
within the framework of [ILENIA Project](https://proyectoilenia.es/) with reference 2022/TL22/00215337, 2022/TL22/00215336, 2022/TL22/00215335, 2022/TL22/00215334.
|
193 |
+
|
194 |
+
### Acknowledgements
|
195 |
+
|
196 |
+
This project benefited from the contributions of many teams and institutions, including:
|
197 |
+
Senado de España, Parlament de Catalunya, Òmnium Cultural, Dialnet, Institut d’Estudis Aranesos,
|
198 |
+
Fundación Elcano, Universidad de Las Palmas de Gran Canaria, Occiglot, Common Crawl, the Welsh Government,
|
199 |
+
the German Research Center for Artificial Intelligence (DFKI) and the partners of Proyecto ILENIA.
|
200 |
+
Their valuable efforts have been instrumental in the development of this work.
|
201 |
+
|
202 |
+
A special acknowledgment is reserved for the NVIDIA Team with whom we have been meeting on a regular basis.
|
203 |
+
Their consistent support has been particularly appreciated throughout the process.
|
204 |
+
|
205 |
+
### Disclaimer
|
206 |
+
Be aware that the model may contain biases or other unintended distortions.
|
207 |
+
When third parties deploy systems or provide services based on this model, or use the model themselves,
|
208 |
+
they bear the responsibility for mitigating any associated risks and ensuring compliance with applicable regulations,
|
209 |
+
including those governing the use of Artificial Intelligence.
|
210 |
+
|
211 |
+
The Barcelona Supercomputing Center, as the owner and creator of the model, shall not be held liable for any outcomes resulting from third-party use.
|
212 |
+
|
213 |
+
## Citation
|
214 |
+
<span style="color:red">Work in progress, paper coming soon.</span>
|
215 |
+
```bibtext
|
216 |
+
@article{salamandra,
|
217 |
+
title={Salamandra Technical Report},
|
218 |
+
author={LangTech@BSC},
|
219 |
+
year={2024},
|
220 |
+
url = {}
|
221 |
+
}
|
222 |
+
```
|
223 |
+
|
224 |
+
## License
|
225 |
+
[Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0)
|
226 |
+
|
227 |
+
## Model Index
|
228 |
+
|Model|Base|Instruct|
|
229 |
+
|:---:|:---:|:---:|
|
230 |
+
|2B| WiP | WiP |
|
231 |
+
|7B| [Link](https://huggingface.co/projecte-aina/salamandra-7b) | [Link](https://huggingface.co/projecte-aina/salamandra-7b-instruct) |
|
232 |
+
|40B| WiP | WiP |
|