---
language:
- pt
---

Sabiá-7B is Portuguese language model developed by [Maritaca AI](https://www.maritaca.ai/).

**Input:** The model accepts only text input.

**Output:** The Model generates text only.

**Model Architecture:** Sabiá-7B 2 is an auto-regressive language model that uses the same architecture of LLaMA-1-7B.

**Tokenizer:** It uses the same tokenizer as LLaMA-1-7B.

**Maximum sequence length:** 2048 tokens.

**Pretraining data:** The model was pretrained on 7 billion tokens from the Portuguese subset of ClueWeb22, starting with the weights of LLaMA-1-7B and further trained for an additional 10 billion tokens, approximately 1.4 epochs of the training dataset.

**Data Freshness:** The pretraining data has a cutoff of mid-2022.

**License:** The licensing is the same as LLaMA-1's, restricting the model's use to research purposes only.

**Paper:** For more details, please refer to our paper: [Sabiá: Portuguese Large Language Models](https://arxiv.org/pdf/2304.07880.pdf) 

Given that Sabiá-7B was trained solely on a language modeling objective without fine-tuning for instruction following, it is recommended for few-shot tasks rather than zero-shot tasks.

**Results** 

Below we show the results on the Poeta benchmark, which consists of 14 Portuguese datasets:

|Model | NPM |
|--|--|
|LLaMA-1-7B| 33.0|
|LLaMA-2-7B| 43.7|
|Sabiá-7B| 48.5|

For more information on the Normalized Preferred Metric (NPM), please check out our paper.

Please use the following bibtex to cite our paper: 
```
@InProceedings{10.1007/978-3-031-45392-2_15,
    author="Pires, Ramon
    and Abonizio, Hugo
    and Almeida, Thales Sales
    and Nogueira, Rodrigo",
    editor="Naldi, Murilo C.
    and Bianchi, Reinaldo A. C.",
    title="Sabi{\'a}: Portuguese Large Language Models",
    booktitle="Intelligent Systems",
    year="2023",
    publisher="Springer Nature Switzerland",
    address="Cham",
    pages="226--240",
    isbn="978-3-031-45392-2"
}
```