README.md

---
license: bigscience-bloom-rail-1.0
language:
- en
- sv
---

# Model Summary

This is a base causal model extended from [bigscience/bloomz-3b](https://huggingface.co/bigscience/bloomz-3b). 

* Model size: 3.02B (~20M more than the base model)
* The tokenizer is extended to support Swedish language. Additional 8068 of tokens trained from Swedish Wiki and OSCAR have been added. The embedding layer is therefore extended too.
* The embedding layer and self-attention query_key_value layers are re-trained on mixed English and Swedish corpuses.

# Intended Use
This model is being created in order to enable **using Swedish and English on LLMs** to cover public research and business use cases. LLMs are intended to be used for language generation or as a pretrained base model. 
**It needs to be further fine-tuned for specific tasks.**

The model inherits bigscience-bloom-rail-1.0 license from the base model. It shall **NOT** be used in bad purposes. For use restrictions, please check out [RAIL License, Use Restrictions](https://huggingface.co/spaces/bigscience/license) Appendix A.

# Training Corpuses:

The model is re-trained with ~800M Swedish tokens and ~260M English tokens.

  * [olm/wikipedia](https://huggingface.co/datasets/olm/wikipedia)
  * [oscar](https://huggingface.co/datasets/oscar)
  * [sbx/superlim-2](https://huggingface.co/datasets/sbx/superlim-2)
  * [Gabriel/xsum_swe](https://huggingface.co/datasets/Gabriel/xsum_swe)
   
# Notes:
* Since the model is only re-trained with Swedish and English. It seems only Swedish and English capabilities are retained. If you want to re-enable the capabilities of other languages, you will need to re-train it with the specific language.
* You might notice the base model is bloomz-3b, which is a intruction fine-tuned version of bloom. After this re-train, it seems to lose the instruction capability as well. So it is now simply a base causal model which could speak Swedish.