File size: 6,172 Bytes
6300fb0 c45b159 6300fb0 c45b159 3c3d283 c45b159 a1f3301 c45b159 005180b c45b159 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 |
---
license: mit
language:
- it
---
--------------------------------------------------------------------------------------------------
<body>
<span class="vertical-text" style="background-color:lightgreen;border-radius: 3px;padding: 3px;"> </span>
<br>
<span class="vertical-text" style="background-color:orange;border-radius: 3px;padding: 3px;"> Task: CHAT</span>
<br>
<span class="vertical-text" style="background-color:lightblue;border-radius: 3px;padding: 3px;"> Model: DIABLO 🔥</span>
<br>
<span class="vertical-text" style="background-color:tomato;border-radius: 3px;padding: 3px;"> Lang: IT</span>
<br>
<span class="vertical-text" style="background-color:lightgrey;border-radius: 3px;padding: 3px;"> </span>
<br>
<span class="vertical-text" style="background-color:#CF9FFF;border-radius: 3px;padding: 3px;"> </span>
</body>
--------------------------------------------------------------------------------------------------
<h3>Model description</h3>
This model is a <b>conversational</b> language model for the <b>Italian</b> language, based on a GPT-like <b>[1]</b> architecture (more specifically, the model has been obtained by modifying Meta's XGLM architecture <b>[2]</b> and exploiting its 1.7B checkpoint).
The model has been trained on a corpus of \~50K Italian conversational exchanges for \~3 epochs (\~15K steps with a batch size of 10), using 3 different learning rates (1e-5, 2e-6, 1e-6) and exploiting FP16 quantization to manage the considerable size of the model.
The training corpus has been built by using Meta's Blenderbot <b>[3]</b> to generate 50K conversational exchanges in English, and then translating them to the Italian language using a machine translation model.
The current release is designed for brief and informal conversations (small talk) covering light topics (mainly food, entertainment and holidays), but several generalizations and improvements will be introduced in future releases.
<h3>Example</h3>
This is an example of intended use of the model, for brief and informal conversations:
<h3>Quick usage</h3>
In order to use the model for inference, the following pipeline is needed:
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
import re
tokenizer = AutoTokenizer.from_pretrained("osiria/diablo-italian-chatbot-1.3b")
model = AutoModelForCausalLM.from_pretrained("osiria/diablo-italian-chatbot-1.3b")
device = torch.device("cpu")
model = model.to(device)
model.eval()
class Diablo:
def __init__(self, tokenizer, model):
self.tokenizer = tokenizer
self.model = model
def _check_sublist(self, lst, sub_lst, sep = " "):
l_type = type(lst[0])
lst = sep.join(list(map(str, lst)))
sub_lst = sep.join(list(map(str, sub_lst)))
return sub_lst in lst
def _exclude_sublist(self, lst, sub_lst, sep = " "):
l_type = type(lst[0])
lst = sep.join(list(map(str, lst)))
sub_lst = sep.join(list(map(str, sub_lst)))
lst = re.sub("\s+", " ", lst.replace(sub_lst, "")).strip().split(sep)
lst = list(map(l_type, lst))
return lst
def generate(self, prompt, sep = "|", max_tokens = 100, excluded = [[40, 19]],
lookback = 1, stop_tokens = [5, 27, 33], sample = False, top_k = 3):
tokens = tokenizer.encode(prompt + sep)
tokens_generated = []
while tokens[-1] not in stop_tokens and len(tokens) < max_tokens:
output = model.forward(input_ids=torch.tensor([tokens]).to(device)).logits[0,-1]
output = torch.softmax(output, dim = 0)
candidates = torch.topk(output, k = top_k)
if sample:
indices = candidates.indices
scores = candidates.values
next_token = indices[torch.multinomial(scores, 1)[0].item()]
else:
next_token = candidates.indices[0]
next_token = next_token.item()
sub_tokens = tokens_generated[-lookback:] + [next_token]
if len(tokens_generated) >= (lookback + 1) and next_token in tokens_generated[-(lookback + 1):]:
next_token = candidates.indices[1]
next_token = next_token.item()
elif len(tokens_generated) >= lookback and self._check_sublist(tokens_generated, sub_tokens):
next_token = candidates.indices[1]
next_token = next_token.item()
tokens = tokens + [next_token]
tokens_generated = tokens_generated + [next_token]
for ex_lst in excluded:
tokens = self._exclude_sublist(tokens, ex_lst)
output = tokenizer.decode(tokens, skip_special_tokens=True)
output = output.split(sep)[-1].strip()
output = output[0].upper() + output[1:]
if output[-1] == tokenizer.decode(stop_tokens[0]):
output = output[:-1]
return output
diablo = Diablo(tokenizer = tokenizer, model = model)
prompt = "Ciao, come stai?"
# setting "sample = True" the model will be more creative but occasionally less accurate
print("OUTPUT:", diablo.generate(prompt, sample = False))
# OUTPUT: Sto bene, grazie
```
<h3>Limitations</h3>
This model has been mainly trained on machine-translated (and synthetic) conversational data, so it might behave erratically when presented with prompts which are too far away from its training set.
Moreover, the heterogeneous nature of the pretraining dataset, together with the limits of the conversational data, might lead the model to produce biased or offensive content with respect to gender, race, ideologies, and political or religious beliefs.
These limitations imply that the model and its outputs should be used with caution, and should not be involved in situations that require the generated text to be fair or true.
<h3>References</h3>
[1] https://arxiv.org/abs/2005.14165
[2] https://arxiv.org/abs/2112.10668
[3] https://arxiv.org/pdf/2004.13637.pdf
<h3>License</h3>
The model is released under <b>MIT</b> license
|