|
--- |
|
license: mit |
|
language: |
|
- it |
|
--- |
|
-------------------------------------------------------------------------------------------------- |
|
|
|
<body> |
|
<span class="vertical-text" style="background-color:lightgreen;border-radius: 3px;padding: 3px;"> </span> |
|
<br> |
|
<span class="vertical-text" style="background-color:orange;border-radius: 3px;padding: 3px;"> Task: CHAT</span> |
|
<br> |
|
<span class="vertical-text" style="background-color:lightblue;border-radius: 3px;padding: 3px;"> Model: DIABLO 🔥</span> |
|
<br> |
|
<span class="vertical-text" style="background-color:tomato;border-radius: 3px;padding: 3px;"> Lang: IT</span> |
|
<br> |
|
<span class="vertical-text" style="background-color:lightgrey;border-radius: 3px;padding: 3px;"> </span> |
|
<br> |
|
<span class="vertical-text" style="background-color:#CF9FFF;border-radius: 3px;padding: 3px;"> </span> |
|
</body> |
|
|
|
-------------------------------------------------------------------------------------------------- |
|
|
|
<h3>Model description</h3> |
|
|
|
This model is a <b>conversational</b> language model for the <b>Italian</b> language, based on a GPT-like <b>[1]</b> architecture (more specifically, the model has been obtained by modifying Meta's XGLM architecture <b>[2]</b> and exploiting its 1.7B checkpoint). |
|
|
|
The model has been trained on a corpus of \~50K Italian conversational exchanges for \~3 epochs (\~15K steps with a batch size of 10), using 3 different learning rates (1e-5, 2e-6, 1e-6) and exploiting FP16 quantization to manage the considerable size of the model. |
|
The training corpus has been built by using Meta's Blenderbot <b>[3]</b> to generate 50K conversational exchanges in English, and then translating them to the Italian language using a machine translation model. |
|
|
|
The current release is designed for brief and informal conversations (small talk) covering light topics (mainly food, entertainment and holidays), but several generalizations and improvements will be introduced in future releases. |
|
|
|
<h3>Example</h3> |
|
|
|
This is an example of intended use of the model, for brief and informal conversations: |
|
|
|
 |
|
|
|
<h3>Quick usage</h3> |
|
|
|
In order to use the model for inference, the following pipeline is needed: |
|
|
|
```python |
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
import torch |
|
import re |
|
|
|
tokenizer = AutoTokenizer.from_pretrained("osiria/diablo-italian-chatbot-1.3b") |
|
model = AutoModelForCausalLM.from_pretrained("osiria/diablo-italian-chatbot-1.3b") |
|
device = torch.device("cpu") |
|
model = model.to(device) |
|
model.eval() |
|
|
|
class Diablo: |
|
|
|
def __init__(self, tokenizer, model): |
|
self.tokenizer = tokenizer |
|
self.model = model |
|
|
|
def _check_sublist(self, lst, sub_lst, sep = " "): |
|
|
|
l_type = type(lst[0]) |
|
lst = sep.join(list(map(str, lst))) |
|
sub_lst = sep.join(list(map(str, sub_lst))) |
|
|
|
return sub_lst in lst |
|
|
|
def _exclude_sublist(self, lst, sub_lst, sep = " "): |
|
|
|
l_type = type(lst[0]) |
|
lst = sep.join(list(map(str, lst))) |
|
sub_lst = sep.join(list(map(str, sub_lst))) |
|
lst = re.sub("\s+", " ", lst.replace(sub_lst, "")).strip().split(sep) |
|
lst = list(map(l_type, lst)) |
|
|
|
return lst |
|
|
|
def generate(self, prompt, sep = "|", max_tokens = 100, excluded = [[40, 19]], |
|
lookback = 1, stop_tokens = [5, 27, 33], sample = False, top_k = 3): |
|
|
|
tokens = tokenizer.encode(prompt + sep) |
|
tokens_generated = [] |
|
while tokens[-1] not in stop_tokens and len(tokens) < max_tokens: |
|
output = model.forward(input_ids=torch.tensor([tokens]).to(device)).logits[0,-1] |
|
output = torch.softmax(output, dim = 0) |
|
candidates = torch.topk(output, k = top_k) |
|
if sample: |
|
indices = candidates.indices |
|
scores = candidates.values |
|
next_token = indices[torch.multinomial(scores, 1)[0].item()] |
|
else: |
|
next_token = candidates.indices[0] |
|
next_token = next_token.item() |
|
sub_tokens = tokens_generated[-lookback:] + [next_token] |
|
if len(tokens_generated) >= (lookback + 1) and next_token in tokens_generated[-(lookback + 1):]: |
|
next_token = candidates.indices[1] |
|
next_token = next_token.item() |
|
elif len(tokens_generated) >= lookback and self._check_sublist(tokens_generated, sub_tokens): |
|
next_token = candidates.indices[1] |
|
next_token = next_token.item() |
|
tokens = tokens + [next_token] |
|
tokens_generated = tokens_generated + [next_token] |
|
for ex_lst in excluded: |
|
tokens = self._exclude_sublist(tokens, ex_lst) |
|
output = tokenizer.decode(tokens, skip_special_tokens=True) |
|
output = output.split(sep)[-1].strip() |
|
output = output[0].upper() + output[1:] |
|
if output[-1] == tokenizer.decode(stop_tokens[0]): |
|
output = output[:-1] |
|
|
|
return output |
|
|
|
diablo = Diablo(tokenizer = tokenizer, model = model) |
|
|
|
prompt = "Ciao, come stai?" |
|
|
|
# setting "sample = True" the model will be more creative but occasionally less accurate |
|
print("OUTPUT:", diablo.generate(prompt, sample = False)) |
|
|
|
# OUTPUT: Sto bene, grazie |
|
``` |
|
|
|
|
|
<h3>Limitations</h3> |
|
|
|
This model has been mainly trained on machine-translated (and synthetic) conversational data, so it might behave erratically when presented with prompts which are too far away from its training set. |
|
Moreover, the heterogeneous nature of the pretraining dataset, together with the limits of the conversational data, might lead the model to produce biased or offensive content with respect to gender, race, ideologies, and political or religious beliefs. |
|
These limitations imply that the model and its outputs should be used with caution, and should not be involved in situations that require the generated text to be fair or true. |
|
|
|
<h3>References</h3> |
|
|
|
[1] https://arxiv.org/abs/2005.14165 |
|
|
|
[2] https://arxiv.org/abs/2112.10668 |
|
|
|
[3] https://arxiv.org/pdf/2004.13637.pdf |
|
|
|
<h3>License</h3> |
|
|
|
The model is released under <b>MIT</b> license |
|
|
