NbAiLab/nb-llama-3.2-3B-Instruct

Model Card: NB-Llama-3.2-3B-Instruct

---

Model Overview

NB-Llama-3.2-3B-Instruct is part of the NB-Llama-3.2 series of models, trained on top of Llama-3.2-3B. This multilingual generative model was fine-tuned specifically to support Norwegian Bokmål, Norwegian Nynorsk, and English, with partial support for Swedish and Danish.

The Instruct-model is trained using Supervised FineTuning (SFT) and then Direct Preference Optimalisation (DPO). The SFT training is based on synthetic datasets, the English Magpie dataset and a translated/filtered version of this dataset. The DPO training is based on Anthropics Helpful and Harmless dataset. The training is supposed to be fairly basic, giving the models a decent undertstanding of a chat template.

The basic idea with this model series was to explore how current state-of-the-art models could be improved for Norwegian by training only on publicly available data. While these models are trained by the National Library of Norway, they do not include data only available through legal deposit. They do, however, contain public data like governmental reports that are both publicly available and legally deposited.

---

Key Features

• Base Model: Built on Llama-3.2-3B.
• Languages:
  • Full support: Norwegian Bokmål (nb), Norwegian Nynorsk (nn), English (en).
  • Partial support: Swedish (sv), Danish (da).
• Purpose: Supports Norwegian-specific tasks such as question-answering, summarization, and language modeling, while being capable of multilingual generation and translation. Efforts have been made to preserve the English capabilities from the underlying Meta Llama model.
• Training Data: Combines publicly available multilingual datasets with synthetic data generation, focusing on Norwegian, English, Swedish, and Danish sources. Additional details are provided below.
• Architecture: The model uses the Llama 3.2 architecture. It is an auto-regressive language model with an optimized transformer architecture. The tuned versions use supervised fine-tuning (SFT) and Direct Preference Optimization (DPO) for alignment.

---

Model Details

• Developer: National Library of Norway (NB-AiLab).
• Parameters: 3 billion.
• Knowledge Cutoff: May 2024.
• License: Llama 3.2 Community License.

---

Motivation

The primary goal of NB-Llama-3.2-3B-Instruct is to advance support for Norwegian language technologies and strengthen support for Norwegian Bokmål and Norwegian Nynorsk. Since much knowledge and culture are also expressed in English, Swedish, and Danish, open sources in these languages are included in the training datasets when possible.

---

Intended Use

Use Cases

• Dialogue systems.
• General multilingual text generation and language modeling.
• Norwegian-specific tasks such as:
  • Summarization of texts in Bokmål or Nynorsk.
  • Question-answering tailored to Norwegian cultural and linguistic contexts.

Out-of-Scope

• Use in violation of applicable laws or regulations.
• Tasks outside the supported languages without additional fine-tuning.
• High-risk domains without appropriate safety measures.

---

How to Use

Please note tht this is still a research project, and the purpose of releasing the models are to investigate the potential in adapting these models for Norwegian language. The intented use case is experiemental. For end-users, we strongly recommend using the instruction-tuned models. We provide quantized models with close to the same accuracy that will run much faster on most platforms. When fine-tuning the instruction-tuned models, best results are obtained when applying the appropriate templates from Llama 3.2.

Using transformers

import torch
from transformers import pipeline

model_id = "NbAiLab/nb-llama-3.2-3B-Instruct"
pipe = pipeline(
    "text-generation",
    model=model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
messages = [
    {"role": "user", "content": "Hvem er du?"},
]
outputs = pipe(
    messages,
    max_new_tokens=256,
)
print(outputs[0]["generated_text"][-1])

---

Training Data

Overview:

The training data is based entirely on publicly available datasets and synthetically generated data. A key aspect of the training process was leveraging high-quality knowledge sources in Norwegian, English, Swedish, and Danish.

For SFT and DPO the following data were used:

• Magpie
• Anthropics Helpful and Harmless
• Various synthetic and translated datasets
• See Llama-3.2-3B for more details on the pretraining data and data selection

---

Licensing

The model is released under the Llama 3.2 Community License, allowing for research and commercial use within defined limitations. Refer to the Acceptable Use Policy for specific restrictions.

---

Citing & Authors

The model was trained and documentation written by Per Egil Kummervold.

Funding and Acknowledgement

Training this model was supported by Google’s TPU Research Cloud (TRC), which generously supplied us with Cloud TPUs essential for our computational needs.