metadata

license: apache-2.0
datasets:
  - nicholasKluge/fine-tuning-instruct-aira
  - Dahoas/synthetic-instruct-gptj-pairwise
  - databricks/databricks-dolly-15k
  - HuggingFaceH4/instruction-dataset
language:
  - en
metrics:
  - bleu
library_name: transformers
tags:
  - alignment
  - instruction tuned
  - text generation
  - conversation
  - assistant
pipeline_tag: text-generation
widget:
  - text: <|startoftext|>Olá! Qual o seu nome?<|endoftext|>
    example_title: Olá
  - text: >-
      <|startoftext|>Você pode me explicar o que é aprendizagem de
      máquina?<|endoftext|>
    example_title: Aprendizagem de máquina
  - text: >-
      <|startoftext|>Você sabe alguma coisa sobre ética das
      virtudes<|endoftext|>
    example_title: Ética das virtudes
  - text: <|startoftext|>O que posso fazer para alegrar minha namorada?<|endoftext|>
    example_title: Conselho
inference:
  parameters:
    temperature: 0.7
    top_k: 50
    max_length: 200

Aira-Instruct-PT-124M (Portuguese)

Aira-Instruct-PT-124M is a instruction-tuned GPT-style model based on GPT-2. The model was trained with a dataset composed of prompt, completions, generated via the Self-Instruct framework. Aira-Instruct-PT-124M instruction-tuning was achieved via conditional text generation.

The dataset used to train this model combines two main sources of data: the synthetic-instruct-gptj-pairwise dataset and a subset of Aira's fine-tuning dataset focused on Ethics, AI, AI safety, and related topics. The dataset is available in both Portuguese and English.

Check our gradio-demo in Spaces.

Details

Size: 124,441,344 total parameters
Dataset: Instruct-Aira Dataset
Language: Portuguese
Number of Epochs: 5
Batch size: 32
Optimizer: torch.optim.AdamW (warmup_steps = 1e2, learning_rate = 5e-4, epsilon = 1e-8)
GPU: 1 NVIDIA A100-SXM4-40GB
Emissions: 0.15 KgCO2
Total Energy Consumption: 0.45 kWh

Epoch/Loss	Training	Validation
1	0.932626	0.767844
2	0.728739	0.723823
3	0.649202	0.705316
4	0.589048	0.698928
5	0.542641	0.700216

Note: This repository has the notebook used to train this model.

Usage

Two special tokens are used to mark the user side of the interaction and the model's response:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

tokenizer = AutoTokenizer.from_pretrained('nicholasKluge/Aira-Instruct-PT-124M')
aira = AutoModelForCausalLM.from_pretrained('nicholasKluge/Aira-Instruct-PT-124M')

aira.eval()
aira.to(device)

question =  input("Enter your question: ")

inputs = tokenizer(tokenizer.bos_token + question + tokenizer.eos_token, return_tensors="pt").to(device)

responses = aira.generate(**inputs,
    bos_token_id=tokenizer.bos_token_id,
    pad_token_id=tokenizer.pad_token_id,
    eos_token_id=tokenizer.eos_token_id,
    do_sample=True,
    top_k=50,
    max_length=200,
    top_p=0.95,
    temperature=0.7,
    num_return_sequences=2)

print(f"Question: 👤 {question}\n")

for i, response in  enumerate(responses):
    # print only the response and remove the question
    print(f'Response {i+1}: 🤖 {tokenizer.decode(response, skip_special_tokens=True).replace(question, "")}')

The model will output something like:

>>> Question: 👤 Olá! Como você se chama?

>>>Response 1: 🤖 Olá! Meu nome é Aira e sou um chatbot projetado para conversar sobre Ética e Segurança da IA. Se você precisar de ajuda com um assunto diferente, por favor, peça "ajuda".
>>>Response 2: 🤖 Olá! Meu nome é Aira e sou um chatbot treinado para responder perguntas sobre Ética e Segurança da IA. Se você precisar de ajuda para navegar em nossa conversa, não hesite em pedir ajuda.

Limitations

🤥 Generative models can perpetuate the generation of pseudo-informative content, that is, false information that may appear truthful. For example, multi-modal generative models can be used to create images with untruthful content, while language models for text generation can automate the generation of misinformation.

🤬 In certain types of tasks, generative models can generate toxic and discriminatory content inspired by historical stereotypes against sensitive attributes (for example, gender, race, and religion). Unfiltered public datasets may also contain inappropriate content, such as pornography, racist images, and social stereotypes, which can contribute to unethical biases in generative models. Furthermore, when prompted with non-English languages, some generative models may perform poorly.

Cite as 🤗


@misc{nicholas22aira,
  doi = {10.5281/zenodo.6989727},
  url = {https://huggingface.co/nicholasKluge/Aira-Instruct-PT-124M},
  author = {Nicholas Kluge Corrêa and Carolina Del Pino},
  title = {Aira},
  year = {2023},
  publisher = {HuggingFace},
  journal = {HuggingFace repository},
}

License

The Aira-Instruct-PT-124M is licensed under the Apache License, Version 2.0. See the LICENSE file for more details.