Leonard Püttmann commited on
Commit
8219f8b
·
verified ·
1 Parent(s): de48f4d

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +70 -3
README.md CHANGED
@@ -1,3 +1,70 @@
1
- ---
2
- license: apache-2.0
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: transformers
3
+ tags:
4
+ - seq2seq
5
+ license: apache-2.0
6
+ datasets:
7
+ - Helsinki-NLP/europarl
8
+ - Helsinki-NLP/opus-100
9
+ language:
10
+ - en
11
+ - it
12
+ base_model:
13
+ - google/t5-efficient-tiny
14
+ pipeline_tag: translation
15
+ ---
16
+
17
+ ## 🍃 Foglietta - A super tiny model for English -> Italian translation
18
+
19
+ Foglietta is an encoder-decoder transformer model for English-Italian text translation based on `google/t5-efficient-tiny`. It was trained on the `en-it` section of `Helsinki-NLP/opus-100` and `Helsinki-NLP/europarl`.
20
+
21
+
22
+ ## Usage
23
+
24
+ ```python
25
+ from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
26
+
27
+ # Load model and tokenizer from checkpoint directory
28
+ tokenizer = AutoTokenizer.from_pretrained("LeonardPuettmann/Foglietta-mt-en-it")
29
+ model = AutoModelForSeq2SeqLM.from_pretrained("LeonardPuettmann/Foglietta-mt-en-it")
30
+
31
+ def generate_response(input_text):
32
+ input_ids = tokenizer("translate English to Italian:" + input_text, return_tensors="pt").input_ids
33
+ output = model.generate(input_ids, max_new_tokens=256)
34
+ return tokenizer.decode(output[0], skip_special_tokens=True)
35
+
36
+ text_to_translate = "I would like a cup of green tea, please."
37
+ response = generate_response(text_to_translate)
38
+ print(response)
39
+ ```
40
+
41
+ As this model is trained on translating sentence pairs, it is best to split longer text into individual sentences, ideally using SpaCy. You can then translate the sentences and join the translations at the end like this:
42
+ ```python
43
+ from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
44
+ import spacy
45
+ # First, install spaCy and the English language model if you haven't already
46
+ # !pip install spacy
47
+ # !python -m spacy download en_core_web_sm
48
+
49
+ nlp = spacy.load("en_core_web_sm")
50
+
51
+ tokenizer = AutoTokenizer.from_pretrained("LeonardPuettmann/Foglietta-mt-en-it")
52
+ model = AutoModelForSeq2SeqLM.from_pretrained("LeonardPuettmann/Foglietta-mt-en-it")
53
+
54
+ def generate_response(input_text):
55
+ input_ids = tokenizer("translate Italian to English: " + input_text, return_tensors="pt").input_ids
56
+ output = model.generate(input_ids, max_new_tokens=256)
57
+ return tokenizer.decode(output[0], skip_special_tokens=True)
58
+
59
+ text = "How are you doing? Today is a beautiful day. I hope you are doing fine."
60
+ doc = nlp(text)
61
+ sentences = [sent.text for sent in doc.sents]
62
+
63
+ sentence_translations = []
64
+ for i, sentence in enumerate(sentences):
65
+ sentence_translation = generate_response(sentence)
66
+ sentence_translations.append(sentence_translation)
67
+
68
+ full_translation = " ".join(sentence_translations)
69
+ print(full_translation)
70
+ ```