Create README.md
Browse files
README.md
ADDED
@@ -0,0 +1,37 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
# Turkish-question-paraphrase-generator
|
2 |
+
mT5 based pre-trained <a href="https://huggingface.co/secometo/mt5-base-turkish-question-paraphrase-generator">model</a> to generate question paraphrases in Turkish language.
|
3 |
+
|
4 |
+
|
5 |
+
## Acknowledgement
|
6 |
+
In this project, which we undertook as an interim project of Yildiz Technical University, our goal was to conduct research on Turkish in area that has not been studied much. In this process, we compared the models trained with different algorithms. Developed a dataset and shared it by writing article for our model. We would like to thank our mentor teacher <a href="https://github.com/mfatihamasyali">Mehmet Fatih Amasyali</a> who has always supported us on this path.
|
7 |
+
|
8 |
+
## Usage
|
9 |
+
|
10 |
+
```python
|
11 |
+
import transformers, sentencepiece
|
12 |
+
|
13 |
+
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
|
14 |
+
tokenizer = AutoTokenizer.from_pretrained("secometo/mt5-base-turkish-question-paraphrase-generator")
|
15 |
+
model = AutoModelForSeq2SeqLM.from_pretrained("secometo/mt5-base-turkish-question-paraphrase-generator")
|
16 |
+
|
17 |
+
original_sentence = tokenizer.encode_plus("Ülke genelinde bisiklet kullanımının artması hakkında ne düşünüyorsun?", return_tensors='pt')
|
18 |
+
paraphrased_sentences = model.generate(original_sentence['input_ids'], max_length=150, num_return_sequences=5, num_beams=5)
|
19 |
+
tokenizer.batch_decode(paraphrased_sentences, skip_special_tokens=True)
|
20 |
+
```
|
21 |
+
## Outputs
|
22 |
+
|
23 |
+
```
|
24 |
+
['ülke genelinde bisiklet kullanımının artması ile ilgili düşünceniz nedir?',
|
25 |
+
'ülke genelinde bisiklet kullanımının artması hakkında düşünceniz nedir?',
|
26 |
+
'ülke genelinde bisiklet kullanımının artması için ne düşünüyorsunuz?',
|
27 |
+
'ülke genelinde bisiklet kullanımının artması hakkında ne düşünüyorsunuz?',
|
28 |
+
'ülke genelinde bisiklet kullanımının artması hakkında fikirleriniz nedir?']
|
29 |
+
```
|
30 |
+
|
31 |
+
|
32 |
+
## Dataset
|
33 |
+
|
34 |
+
We used 50994 question sentence pairs, which are created manually, to train our model. The dataset is provided our mentor. Sentences were extracted from the titles of topics in popular Turkish forum website donanimhaber.com. We augmented the dataset by writing ten thousand sentences per person with <a href="https://github.com/sercaksoy?tab=repositories">Sercan Aksoy</a>.
|
35 |
+
|
36 |
+
|
37 |
+
|