nicholasKluge commited on
Commit
9289255
·
verified ·
1 Parent(s): b8a9d7a

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +431 -0
README.md ADDED
@@ -0,0 +1,431 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - pt
4
+ license: apache-2.0
5
+ library_name: transformers
6
+ tags:
7
+ - text-generation-inference
8
+ datasets:
9
+ - nicholasKluge/instruct-aira-dataset-v3
10
+ - cnmoro/GPT4-500k-Augmented-PTBR-Clean
11
+ - rhaymison/orca-math-portuguese-64k
12
+ - nicholasKluge/reward-aira-dataset
13
+ metrics:
14
+ - perplexity
15
+ pipeline_tag: text-generation
16
+ widget:
17
+ - text: "<instruction>Cite algumas bandas de rock brasileiras famosas.</instruction>"
18
+ example_title: Exemplo
19
+ - text: "<instruction>Invente uma história sobre um encanador com poderes mágicos.</instruction>"
20
+ example_title: Exemplo
21
+ - text: "<instruction>Qual cidade é a capital do estado do Rio Grande do Sul?</instruction>"
22
+ example_title: Exemplo
23
+ - text: "<instruction>Diga o nome de uma maravilha culinária característica da cosinha Portuguesa?</instruction>"
24
+ example_title: Exemplo
25
+ inference:
26
+ parameters:
27
+ repetition_penalty: 1.2
28
+ temperature: 0.2
29
+ top_k: 20
30
+ top_p: 0.2
31
+ max_new_tokens: 150
32
+ co2_eq_emissions:
33
+ emissions: 21890
34
+ source: CodeCarbon
35
+ training_type: pre-training
36
+ geographical_location: Germany
37
+ hardware_used: NVIDIA A100-SXM4-80GB
38
+ model-index:
39
+ - name: Tucano-1b1-Instruct
40
+ results:
41
+ - task:
42
+ type: text-generation
43
+ name: Text Generation
44
+ dataset:
45
+ name: CALAME-PT
46
+ type: NOVA-vision-language/calame-pt
47
+ split: all
48
+ args:
49
+ num_few_shot: 0
50
+ metrics:
51
+ - type: acc
52
+ value: 56.55
53
+ name: accuracy
54
+ source:
55
+ url: https://huggingface.co/datasets/NOVA-vision-language/calame-pt
56
+ name: Context-Aware LAnguage Modeling Evaluation for Portuguese
57
+ - task:
58
+ type: text-generation
59
+ name: Text Generation
60
+ dataset:
61
+ name: LAMBADA-PT
62
+ type: nicholasKluge/LAMBADA-PT
63
+ split: train
64
+ args:
65
+ num_few_shot: 0
66
+ metrics:
67
+ - type: acc
68
+ value: 35.53
69
+ name: accuracy
70
+ source:
71
+ url: https://huggingface.co/datasets/TucanoBR/lambada-pt
72
+ name: LAMBADA-PT
73
+ - task:
74
+ type: text-generation
75
+ name: Text Generation
76
+ dataset:
77
+ name: ENEM Challenge (No Images)
78
+ type: eduagarcia/enem_challenge
79
+ split: train
80
+ args:
81
+ num_few_shot: 3
82
+ metrics:
83
+ - type: acc
84
+ value: 21.06
85
+ name: accuracy
86
+ source:
87
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard
88
+ name: Open Portuguese LLM Leaderboard
89
+ - task:
90
+ type: text-generation
91
+ name: Text Generation
92
+ dataset:
93
+ name: BLUEX (No Images)
94
+ type: eduagarcia-temp/BLUEX_without_images
95
+ split: train
96
+ args:
97
+ num_few_shot: 3
98
+ metrics:
99
+ - type: acc
100
+ value: 26.01
101
+ name: accuracy
102
+ source:
103
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard
104
+ name: Open Portuguese LLM Leaderboard
105
+ - task:
106
+ type: text-generation
107
+ name: Text Generation
108
+ dataset:
109
+ name: OAB Exams
110
+ type: eduagarcia/oab_exams
111
+ split: train
112
+ args:
113
+ num_few_shot: 3
114
+ metrics:
115
+ - type: acc
116
+ value: 26.47
117
+ name: accuracy
118
+ source:
119
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard
120
+ name: Open Portuguese LLM Leaderboard
121
+ - task:
122
+ type: text-generation
123
+ name: Text Generation
124
+ dataset:
125
+ name: Assin2 RTE
126
+ type: assin2
127
+ split: test
128
+ args:
129
+ num_few_shot: 15
130
+ metrics:
131
+ - type: f1_macro
132
+ value: 67.78
133
+ name: f1-macro
134
+ source:
135
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard
136
+ name: Open Portuguese LLM Leaderboard
137
+ - task:
138
+ type: text-generation
139
+ name: Text Generation
140
+ dataset:
141
+ name: Assin2 STS
142
+ type: eduagarcia/portuguese_benchmark
143
+ split: test
144
+ args:
145
+ num_few_shot: 10
146
+ metrics:
147
+ - type: pearson
148
+ value: 8.88
149
+ name: pearson
150
+ source:
151
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard
152
+ name: Open Portuguese LLM Leaderboard
153
+ - task:
154
+ type: text-generation
155
+ name: Text Generation
156
+ dataset:
157
+ name: FaQuAD NLI
158
+ type: ruanchaves/faquad-nli
159
+ split: test
160
+ args:
161
+ num_few_shot: 15
162
+ metrics:
163
+ - type: f1_macro
164
+ value: 43.97
165
+ name: f1-macro
166
+ source:
167
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard
168
+ name: Open Portuguese LLM Leaderboard
169
+ - task:
170
+ type: text-generation
171
+ name: Text Generation
172
+ dataset:
173
+ name: HateBR Binary
174
+ type: ruanchaves/hatebr
175
+ split: test
176
+ args:
177
+ num_few_shot: 25
178
+ metrics:
179
+ - type: f1_macro
180
+ value: 31.28
181
+ name: f1-macro
182
+ source:
183
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard
184
+ name: Open Portuguese LLM Leaderboard
185
+ - task:
186
+ type: text-generation
187
+ name: Text Generation
188
+ dataset:
189
+ name: PT Hate Speech Binary
190
+ type: hate_speech_portuguese
191
+ split: test
192
+ args:
193
+ num_few_shot: 25
194
+ metrics:
195
+ - type: f1_macro
196
+ value: 41.23
197
+ name: f1-macro
198
+ source:
199
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard
200
+ name: Open Portuguese LLM Leaderboard
201
+ - task:
202
+ type: text-generation
203
+ name: Text Generation
204
+ dataset:
205
+ name: tweetSentBR
206
+ type: eduagarcia-temp/tweetsentbr
207
+ split: test
208
+ args:
209
+ num_few_shot: 25
210
+ metrics:
211
+ - type: f1_macro
212
+ value: 22.03
213
+ name: f1-macro
214
+ source:
215
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard
216
+ name: Open Portuguese LLM Leaderboard
217
+ - task:
218
+ type: text-generation
219
+ name: Text Generation
220
+ dataset:
221
+ name: ARC-Challenge (PT)
222
+ type: arc_pt
223
+ args:
224
+ num_few_shot: 25
225
+ metrics:
226
+ - type: acc_norm
227
+ value: 30.77
228
+ name: normalized accuracy
229
+ source:
230
+ url: https://github.com/nlp-uoregon/mlmm-evaluation
231
+ name: Evaluation Framework for Multilingual Large Language Models
232
+ - task:
233
+ type: text-generation
234
+ name: Text Generation
235
+ dataset:
236
+ name: HellaSwag (PT)
237
+ type: hellaswag_pt
238
+ args:
239
+ num_few_shot: 10
240
+ metrics:
241
+ - type: acc_norm
242
+ value: 43.50
243
+ name: normalized accuracy
244
+ source:
245
+ url: https://github.com/nlp-uoregon/mlmm-evaluation
246
+ name: Evaluation Framework for Multilingual Large Language Models
247
+ - task:
248
+ type: text-generation
249
+ name: Text Generation
250
+ dataset:
251
+ name: TruthfulQA (PT)
252
+ type: truthfulqa_pt
253
+ args:
254
+ num_few_shot: 0
255
+ metrics:
256
+ - type: mc2
257
+ value: 41.14
258
+ name: bleurt
259
+ source:
260
+ url: https://github.com/nlp-uoregon/mlmm-evaluation
261
+ name: Evaluation Framework for Multilingual Large Language Models
262
+ - task:
263
+ type: text-generation
264
+ name: Text Generation
265
+ dataset:
266
+ name: Alpaca-Eval (PT)
267
+ type: alpaca_eval_pt
268
+ args:
269
+ num_few_shot: 0
270
+ metrics:
271
+ - type: lc_winrate
272
+ value: 8.80
273
+ name: length controlled winrate
274
+ source:
275
+ url: https://github.com/tatsu-lab/alpaca_eval
276
+ name: AlpacaEval
277
+ base_model:
278
+ - TucanoBR/Tucano-1b1
279
+ ---
280
+ # Tucano-1b1-Instruct
281
+
282
+ <img src="./logo.png" alt="An illustration of a Tucano bird showing vibrant colors like yellow, orange, blue, green, and black." height="200">
283
+
284
+ ## Model Summary
285
+
286
+ Tucano-1b1-Instruct is a fine-tuned version of [Tucano-1b1](https://huggingface.co/TucanoBR/Tucano-1b1). **[Tucano](https://huggingface.co/TucanoBR)** is a series of decoder-transformers based on the Llama 2 architecture, pretrained natively in Portuguese. All Tucano models were trained on **[GigaVerbo](https://huggingface.co/datasets/TucanoBR/GigaVerbo)**, a concatenation of deduplicated Portuguese text corpora amounting to 200 billion tokens.
287
+
288
+ The fine-tuning process was divided into two stages:
289
+
290
+ - Supervised fine-tuning (SFT) using the [TucanoBR/Tucano-SFT](https://huggingface.co/datasets/TucanoBR/Tucano-SFT), a concatenation of three different instruction tuning datasets ([`cnmoro/GPT4-500k-Augmented-PTBR-Clean`](https://huggingface.co/datasets/cnmoro/GPT4-500k-Augmented-PTBR-Clean), [`rhaymison/orca-math-portuguese-64k`](https://huggingface.co/datasets/rhaymison/orca-math-portuguese-64k), [`nicholasKluge/instruct-aira-dataset-v3`](https://huggingface.co/datasets/nicholasKluge/instruct-aira-dataset-v3)).
291
+ - Direct Preference Optimization (DPO) using the [nicholasKluge/reward-aira-dataset](https://huggingface.co/datasets/nicholasKluge/reward-aira-dataset).
292
+
293
+ Read our preprint [here](https://arxiv.org/abs/xxxx.xxxxx).
294
+
295
+ ## Details
296
+
297
+ - **Architecture:** a Transformer-based model pre-trained via causal language modeling
298
+ - **Size:** 1,100,048,384 parameters
299
+ - **Context length:** 2048 tokens
300
+ - **Dataset:**
301
+ - [cnmoro/GPT4-500k-Augmented-PTBR-Clean](https://huggingface.co/datasets/cnmoro/GPT4-500k-Augmented-PTBR-Clean)
302
+ - [rhaymison/orca-math-portuguese-64k](https://huggingface.co/datasets/rhaymison/orca-math-portuguese-64k)
303
+ - [nicholasKluge/instruct-aira-dataset-v3](https://huggingface.co/datasets/nicholasKluge/instruct-aira-dataset-v3)
304
+ - [nicholasKluge/reward-aira-dataset](https://huggingface.co/datasets/nicholasKluge/reward-aira-dataset)
305
+ - **Language:** Portuguese
306
+ - **Training time**: ~ 12 hours
307
+ - **Emissions:** 22 KgCO2 (Germany)
308
+ - **Total energy consumption:** 58 kWh
309
+
310
+ This repository has the [source code](https://github.com/Nkluge-correa/Tucano) used to train this model. The main libraries used are:
311
+
312
+ - [PyTorch](https://github.com/pytorch/pytorch)
313
+ - [Transformers](https://github.com/huggingface/transformers)
314
+ - [Datasets](https://github.com/huggingface/datasets)
315
+ - [Tokenizers](https://github.com/huggingface/tokenizers)
316
+ - [Sentencepiece](https://github.com/google/sentencepiece)
317
+ - [Accelerate](https://github.com/huggingface/accelerate)
318
+ - [FlashAttention](https://github.com/Dao-AILab/flash-attention)
319
+ - [Liger Kernel](https://github.com/linkedin/Liger-Kernel)
320
+ - [Codecarbon](https://github.com/mlco2/codecarbon)
321
+ - [TRL](https://github.com/huggingface/trl)
322
+
323
+ ## Intended Uses
324
+
325
+ The primary intended use of the Tucano models is to serve as foundations for research and development involving native Portuguese language modeling. Checkpoints saved during training are designed to provide a controlled setting for performing comparative experiments, specifically regarding the effects of active pretraining on the performance of currently available benchmarks. You may also fine-tune and adapt Tucano models for deployment if your use follows the Apache 2.0 license. If you decide to use the Tucano models as a basis for your fine-tuned model, please conduct your own risk and bias assessment.
326
+
327
+ ## Out-of-scope Use
328
+
329
+ - Tucano models are **not intended for deployment**. They are not an out-of-the-box product and should not be used for human-facing interactions.
330
+
331
+ - Tucano models are for **the Portuguese language only** and are unsuitable for text generation tasks in other languages.
332
+
333
+ - Tucano models have **not been fine-tuned** for downstream tasks.
334
+
335
+ ## Basic usage
336
+
337
+ Using the `pipeline`:
338
+
339
+ ```python
340
+ from transformers import pipeline
341
+
342
+ generator = pipeline("text-generation", model="TucanoBR/Tucano-1b1-Instruct")
343
+
344
+ completions = generator("<instruction>Qual cidade é a capital do estado do Rio Grande do Sul?</instruction>", num_return_sequences=2, max_new_tokens=100)
345
+
346
+ for comp in completions:
347
+ print(f"🤖 {comp['generated_text']}")
348
+ ```
349
+
350
+ Using the `AutoTokenizer` and `AutoModelForCausalLM`:
351
+
352
+ ```python
353
+ from transformers import GenerationConfig, TextGenerationPipeline, AutoTokenizer, AutoModelForCausalLM
354
+ import torch
355
+
356
+ # Specify the model and tokenizer
357
+ model_id = "TucanoBR/Tucano-1b1-Instruct"
358
+ tokenizer = AutoTokenizer.from_pretrained(model_id)
359
+ model = AutoModelForCausalLM.from_pretrained(model_id)
360
+
361
+ # Specify the generation parameters as you like
362
+ generation_config = GenerationConfig(
363
+ **{
364
+ "do_sample": True,
365
+ "max_new_tokens": 2048,
366
+ "renormalize_logits": True,
367
+ "repetition_penalty": 1.2,
368
+ "temperature": 0.3,
369
+ "top_k": 30,
370
+ "top_p": 0.3,
371
+ "use_cache": True,
372
+ }
373
+ )
374
+
375
+ device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
376
+ generator = TextGenerationPipeline(model=model, task="text-generation", tokenizer=tokenizer, device=device)
377
+
378
+ # Generate text
379
+ prompt = "<instruction>Qual cidade é a capital do estado do Rio Grande do Sul?</instruction>"
380
+ completion = generator(prompt, generation_config=generation_config)
381
+ print(completion[0]['generated_text'])
382
+ ```
383
+
384
+ ## Limitations
385
+
386
+ Like almost all other language models trained on large text datasets scraped from the web, the Tucano models show behavior that does not make them an out-of-the-box solution to many real-world applications, especially those requiring factual, reliable, and nontoxic text generation. Tucano models are all subject to the following:
387
+
388
+ - **Hallucinations:** Tucano models can produce content that can be mistaken as true facts, but are misleading or entirely false, i.e., hallucination.
389
+
390
+ - **Biases and Toxicity:** Tucano models inherit the social and historical stereotypes from the data used to train them. Given these biases, the model can produce toxic content, i.e., harmful, offensive, or detrimental to individuals, groups, or communities.
391
+
392
+ - **Unreliable Code:** Tucano models may produce incorrect code snippets and statements. These code generations should not be treated as suggestions or accurate solutions.
393
+
394
+ - **Language Limitations:** Tucano models are primarily designed to interact with Portuguese. Other languages might challenge its comprehension, leading to potential misinterpretations or errors in response.
395
+
396
+ - **Repetition and Verbosity:** Tucano models may get stuck on repetition loops (especially if the repetition penalty during generations is set to a meager value) or produce verbose responses unrelated to the prompt it was given.
397
+
398
+ Hence, even though our models are released with a permissive license, we urge users to perform their risk analysis on them if they intend to use them for real-world applications. We also have humans moderating the outputs of these models in applications where they will interact with an audience, guaranteeing users are always aware they are interacting with a language model.
399
+
400
+ ## Evaluations
401
+
402
+ To evaluate the `Instruct` versions of our models, we used [AlpacaEval](https://github.com/tatsu-lab/alpaca_eval) 2.0 with length-controlled win-rates, a fast and relatively cheap evaluation method that is highly correlated with human preferences. To learn more about our evaluation read [our documentation](https://github.com/Nkluge-correa/Tucano/blob/main/evaluations/README.md).
403
+
404
+ | | Avg. Length | Wins | Base Wins | Total Matches | Length-Controlled Win Rate (%) | LC Std. Error |
405
+ |-------------------------|-------------|------|-----------|---------------|--------------------------------|---------------|
406
+ | Llama-3.2-3B-Instruct | 1609 | 257 | 548 | 805 | 21.06 | 0.075 |
407
+ | **Tucano-2b4-Instruct** | 1183 | 71 | 734 | 805 | 8.03 | 0.062 |
408
+ | **Tucano-1b1-Instruct** | 1667 | 124 | 681 | 805 | 8.80 | 0.083 |
409
+ | Llama-3.2-1B-Instruct | 1429 | 99 | 706 | 805 | 7.15 | 0.057 |
410
+ | TeenyTinyLlama-460m-Chat| 1333 | 28 | 777 | 805 | 2.84 | 0.059 |
411
+ | Sabiá-7b | 5011 | 1 | 804 | 805 | 0.076 | 0.0043 |
412
+ | Gervásio-7b | 5740 | 1 | 804 | 805 | 0.026 | 0.0016 |
413
+
414
+ ## Cite as 🤗
415
+
416
+ ```latex
417
+ @misc{correa24tucano,
418
+ title = {{Tucano: Advancing Neural Text Generation for Portuguese}},
419
+ author = {Corr{\^e}a, Nicholas Kluge and Sen, Aniket and Falk, Sophia and Fatimah, Shiza},
420
+ journal={arXiv preprint arXiv:xxxx.xxxxx},
421
+ year={2024}
422
+ }
423
+ ```
424
+
425
+ ## Aknowlegments
426
+
427
+ We gratefully acknowledge the granted access to the [Marvin cluster](https://www.hpc.uni-bonn.de/en/systems/marvin) hosted by [University of Bonn](https://www.uni-bonn.de/en) along with the support provided by its High Performance Computing \& Analytics Lab.
428
+
429
+ ## License
430
+
431
+ Tucano is licensed under the Apache License, Version 2.0. For more details, see the [LICENSE](LICENSE) file.