pere commited on
Commit
02ad9cd
1 Parent(s): 01a508f

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +136 -0
README.md ADDED
@@ -0,0 +1,136 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+
3
+ language:
4
+ - no # Generic Norwegian
5
+ - nb # Norwegian Bokmål
6
+ - nn # Norwegian Nynorsk
7
+ - en # English
8
+ - sv # Swedish
9
+ - da # Danish
10
+ tags:
11
+ - norwegian
12
+ - bokmål
13
+ - nynorsk
14
+ - swedish
15
+ - danish
16
+ - multilingual
17
+ - text-generation
18
+ pipeline_tag: text-generation
19
+ license: llama3.2
20
+ ---
21
+
22
+ ## Model Card: NB-Llama-3.2-1B
23
+
24
+ ---
25
+
26
+ ### Model Overview
27
+
28
+ **NB-Llama-3.2-1B** is part of the **NB-Llama-3.2** series of models, trained on top of [Llama-3.2-1B](https://huggingface.co/meta-llama/Llama-3.2-1B). This multilingual generative model was fine-tuned specifically to support Norwegian Bokmål, Norwegian Nynorsk, and English, with partial support for Swedish and Danish.
29
+
30
+ The basic idea with this model series was to explore how current state-of-the-art models could be improved for Norwegian by training only on publicly available data. While these models are trained by the National Library of Norway, they do not include data only available through legal deposit. They do, however, contain public data like governmental reports that are both publicly available and legally deposited.
31
+
32
+ ---
33
+
34
+ ### Key Features
35
+
36
+ - **Base Model**: Built on Llama-3.2-1B.
37
+ - **Languages**:
38
+ - Full support: Norwegian Bokmål (nb), Norwegian Nynorsk (nn), English (en).
39
+ - Partial support: Swedish (sv), Danish (da).
40
+ - **Purpose**: Supports Norwegian-specific tasks such as question-answering, summarization, and language modeling, while being capable of multilingual generation and translation. Efforts have been made to preserve the English capabilities from the underlying Meta Llama model.
41
+ - **Training Data**: Combines publicly available multilingual datasets with synthetic data generation, focusing on Norwegian, English, Swedish, and Danish sources. Additional details are provided below.
42
+ - **Architecture**: The model uses the Llama 3.2 architecture. It is an auto-regressive language model with an optimized transformer architecture. The tuned versions use supervised fine-tuning (SFT) and Direct Preference Optimization (DPO) for alignment.
43
+
44
+ ---
45
+
46
+ ### Model Details
47
+
48
+ - **Developer**: National Library of Norway (NB-AiLab).
49
+ - **Parameters**: 1 billion.
50
+ - **Knowledge Cutoff**: May 2024.
51
+ - **License**: [Llama 3.2 Community License](https://github.com/meta-llama/llama-models/blob/main/models/llama3.2/LICENSE).
52
+
53
+ ---
54
+
55
+ ### Motivation
56
+
57
+ The primary goal of **NB-Llama-3.2-1B** is to advance support for Norwegian language technologies and strengthen support for Norwegian Bokmål and Norwegian Nynorsk. Since much knowledge and culture are also expressed in English, Swedish, and Danish, open sources in these languages are included in the training datasets when possible.
58
+
59
+ ---
60
+
61
+ ### Intended Use
62
+
63
+ #### Use Cases
64
+
65
+ - Dialogue systems.
66
+ - General multilingual text generation and language modeling.
67
+ - Norwegian-specific tasks such as:
68
+ - Summarization of texts in Bokmål or Nynorsk.
69
+ - Question-answering tailored to Norwegian cultural and linguistic contexts.
70
+
71
+ #### Out-of-Scope
72
+
73
+ - Use in violation of applicable laws or regulations.
74
+ - Tasks outside the supported languages without additional fine-tuning.
75
+ - High-risk domains without appropriate safety measures.
76
+
77
+ ---
78
+
79
+ ### How to Use
80
+
81
+ Please note tht this is still a research project, and the purpose of releasing the models are to investigate the potential in adapting these models for Norwegian language. The intented use case is experiemental. For end-users, we strongly recommend using the instruction-tuned models. We provide quantized models with close to the same accuracy that will run much faster on most platforms. When fine-tuning the instruction-tuned models, best results are obtained when applying the appropriate templates from Llama 3.2.
82
+
83
+ #### Using `transformers`
84
+
85
+ ```python
86
+ import transformers
87
+
88
+ model_id = "north/nb-llama-3.2-1B"
89
+
90
+ pipeline = transformers.pipeline(
91
+ "text-generation",
92
+ model=model_id,
93
+ model_kwargs={"torch_dtype": "bfloat16"},
94
+ device_map="auto"
95
+ )
96
+
97
+ output = pipeline("Hva er Nasjonalbibliotekets rolle i AI-utvikling?")
98
+ print(output)
99
+ ```
100
+ ---
101
+
102
+ ### Training Data
103
+
104
+ **Overview:**
105
+
106
+ The training data is based entirely on publicly available datasets and synthetically generated data. A key aspect of the training process was leveraging high-quality knowledge sources in Norwegian, English, Swedish, and Danish.
107
+
108
+ Parts of the following publicly available datasets were used:
109
+
110
+ - [CulturaX](https://huggingface.co/datasets/uonlp/CulturaX)
111
+ - [High Performance Language Technologies (HPLT)](https://huggingface.co/datasets/HPLT/hplt_monolingual_v1_2)
112
+ - [Norwegian Colossal Corpus (NCC)](https://huggingface.co/datasets/NCC/Norwegian-Colossal-Corpus)
113
+ - [Wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia)
114
+
115
+ ---
116
+
117
+ ### Data Selection
118
+
119
+ To ensure the highest quality training data, only a small subset of the original raw data was used. An encoder-only classifier built on [nb-bert-base](https://huggingface.co/NbAiLab/nb-bert-base) was trained to evaluate both educational value and linguistic quality of the training samples.
120
+
121
+ - **Categorization Methods:**
122
+ - Inspired by the [FineWeb](https://example.com/FineWeb) project.
123
+ - Evaluated for:
124
+ - **Educational Value:** Prioritizing high-value training samples.
125
+ - **Linguistic Quality:** Ensuring clarity and accuracy in training data.
126
+ - **Guidance and Release:**
127
+ - Categorization was guided by insights from [Gemini 1.5](https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/#gemini-15).
128
+ - The classifiers are released alongside this model and are [available here](https://classifier-release-link-here).
129
+
130
+ ---
131
+
132
+ ### Licensing
133
+
134
+ The model is released under the [Llama 3.2 Community License](https://github.com/meta-llama/llama-models/blob/main/models/llama3.2/LICENSE), allowing for research and commercial use within defined limitations. Refer to the [Acceptable Use Policy](https://llama.meta.com/llama3.2/use-policy) for specific restrictions.
135
+
136
+ ---