osiria/diablo-italian-chatbot-1.3b

---

 
    Task: CHAT
    Model: DIABLO 🔥
    Lang: IT
  
 

---

Model description

This model is a conversational language model for the Italian language, based on a GPT-like [1] architecture (more specifically, the model has been obtained by modifying Meta's XGLM architecture [2] and exploiting its 1.7B checkpoint).

The model has been trained on a corpus of ~50K Italian conversational exchanges for ~3 epochs (~15K steps with a batch size of 10), using 3 different learning rates (1e-5, 2e-6, 1e-6) and exploiting FP16 quantization to manage the considerable size of the model. The training corpus has been built by using Meta's Blenderbot [3] to generate 50K conversational exchanges in English, and then translating them to the Italian language using a machine translation model.

The current release is designed for brief and informal conversations (small talk) covering light topics (mainly food, entertainment and holidays), but several generalizations and improvements will be introduced in future releases.

Example

This is an example of intended use of the model, for brief and informal conversations:

Quick usage

In order to use the model for inference, the following pipeline is needed:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
import re

tokenizer = AutoTokenizer.from_pretrained("osiria/diablo-italian-chatbot-1.3b")
model = AutoModelForCausalLM.from_pretrained("osiria/diablo-italian-chatbot-1.3b")
device = torch.device("cpu")
model = model.to(device)
model.eval()

class Diablo:
    
    def __init__(self, tokenizer, model):
        self.tokenizer = tokenizer
        self.model = model
        
    def _check_sublist(self, lst, sub_lst, sep = " "):
        
        l_type = type(lst[0])
        lst = sep.join(list(map(str, lst)))
        sub_lst = sep.join(list(map(str, sub_lst)))
        
        return sub_lst in lst
    
    def _exclude_sublist(self, lst, sub_lst, sep = " "):
        
        l_type = type(lst[0])
        lst = sep.join(list(map(str, lst)))
        sub_lst = sep.join(list(map(str, sub_lst)))
        lst = re.sub("\s+", " ", lst.replace(sub_lst, "")).strip().split(sep)
        lst = list(map(l_type, lst))
        
        return lst
        
    def generate(self, prompt, sep = "|", max_tokens = 100, excluded = [[40, 19]], 
                 lookback = 1, stop_tokens = [5, 27, 33], sample = False, top_k = 3):
        
        tokens = tokenizer.encode(prompt + sep)
        tokens_generated = []
        while tokens[-1] not in stop_tokens and len(tokens) < max_tokens:
            output = model.forward(input_ids=torch.tensor([tokens]).to(device)).logits[0,-1]
            output = torch.softmax(output, dim = 0)
            candidates = torch.topk(output, k = top_k)
            if sample:
                indices = candidates.indices
                scores = candidates.values
                next_token = indices[torch.multinomial(scores, 1)[0].item()]
            else:
                next_token = candidates.indices[0]
            next_token = next_token.item()
            sub_tokens = tokens_generated[-lookback:] + [next_token]
            if len(tokens_generated) >= (lookback + 1) and next_token in tokens_generated[-(lookback + 1):]:
                next_token = candidates.indices[1]
                next_token = next_token.item()
            elif len(tokens_generated) >= lookback and self._check_sublist(tokens_generated, sub_tokens):
                next_token = candidates.indices[1]
                next_token = next_token.item()
            tokens = tokens + [next_token]
            tokens_generated = tokens_generated + [next_token]
        for ex_lst in excluded:
            tokens = self._exclude_sublist(tokens, ex_lst)
        output = tokenizer.decode(tokens, skip_special_tokens=True)
        output = output.split(sep)[-1].strip()
        output = output[0].upper() + output[1:]
        if output[-1] == tokenizer.decode(stop_tokens[0]):
            output = output[:-1]
        
        return output
    
diablo = Diablo(tokenizer = tokenizer, model = model)

prompt = "Ciao, come stai?"

# setting "sample = True" the model will be more creative but occasionally less accurate
print("OUTPUT:", diablo.generate(prompt, sample = False))

# OUTPUT: Sto bene, grazie

Limitations

This model has been mainly trained on machine-translated (and synthetic) conversational data, so it might behave erratically when presented with prompts which are too far away from its training set. Moreover, the heterogeneous nature of the pretraining dataset, together with the limits of the conversational data, might lead the model to produce biased or offensive content with respect to gender, race, ideologies, and political or religious beliefs. These limitations imply that the model and its outputs should be used with caution, and should not be involved in situations that require the generated text to be fair or true.

References

[1] https://arxiv.org/abs/2005.14165

[2] https://arxiv.org/abs/2112.10668

[3] https://arxiv.org/pdf/2004.13637.pdf

License

The model is released under MIT license