Update README.md

README.md

@@ -1,3 +1,207 @@
 ---
+language:
+- ru
+tags:
+- spellchecking
+- pytorch
+- natural language generation
 license: mit
+metrics:
+- precision
+- recall
+- f1
+library_name: transformers
+model-index:
+- name: sage-fredt5-large
+  results:
+    - task:
+        type: text-generation
+      dataset:
+        type: spellcheck_benchmark
+        name: RUSpellRU (spell&punct)
+      metrics:
+      - name: F1 (spell)
+        type: f1_spell
+        value: 62.2
+        verified: false
+      - name: F1 (punct)
+        type: f1_punct
+        value: 60.2
+        verified: false
+      - name: F1 (case)
+        type: f1_case
+        value: 78.1
+        verified: false
+    - task:
+        type: text-generation
+      dataset:
+        type: spellcheck_benchmark
+        name: MultidomainGold (spell&punct)
+      metrics:
+      - name: F1 (spell)
+        type: f1_spell
+        value: 46.3
+        verified: false
+      - name: F1 (punct)
+        type: f1_punct
+        value: 21.6
+        verified: false
+      - name: F1 (case)
+        type: f1_case
+        value: 34.0
+        verified: false
+    - task:
+        type: text-generation
+      dataset:
+        type: spellcheck_benchmark
+        name: MedSpellchecker (spell&punct)
+      metrics:
+      - name: F1 (spell)
+        type: f1_spell
+        value: 42.7
+        verified: false
+      - name: F1 (punct)
+        type: f1_punct
+        value: 15.7
+        verified: false
+      - name: F1 (case)
+        type: f1_case
+        value: 41.9
+        verified: false
+    - task:
+        type: text-generation
+      dataset:
+        type: spellcheck_benchmark
+        name: GitHubTypoCorpusRu (spell&punct)
+      metrics:
+      - name: F1 (spell)
+        type: f1_spell
+        value: 46.3
+        verified: false
+      - name: F1 (punct)
+        type: f1_punct
+        value: 20.2
+        verified: false
+      - name: F1 (case)
+        type: f1_case
+        value: 12.6
+        verified: false
 ---
+# sage-fredt5-large
+
+![banner](images/sage_banner.jpg)
+
+## Summary
+
+The model corrects spelling and punctuation errors and typos by bringing all the words in the text to the norm of the Russian language.
+Corrector had been trained based on the model [FRED-T5-large](https://huggingface.co/ai-forever/FRED-T5-large). 
+An extensive dataset with “artificial” errors was taken as a training corpus: the corpus was assembled on the basis of the Russian-language Wikipedia and transcripts of Russian-language videos, then typos and spelling errors were automatically introduced into it using the library [SAGE](https://github.com/ai-forever/sage).
+
+## Public references
+- [SAGE library announcement](https://youtu.be/yFfkV0Qjuu0), DataFest 2023
+- [Paper about synthetic error generation methods](https://www.dialog-21.ru/media/5914/martynovnplusetal056.pdf), Dialogue 2023
+- [SAGE EACL 2024 paper](https://aclanthology.org/2024.findings-eacl.10/)
+
+
+## Examples
+| Input | Output |
+| --- | --- |
+| И не чсно прохожим в этот день непогожйи почему я веселый такйо | И не ясно прохожим в этот день непогожий, почему я веселый такой. |
+| Каждй день воттак делой, и спена балеть нибудет. А вотак каждый день ниделай | Каждый день вот так делай и спина болеть не будет. А вот так каждый день не делай. |
+| Основая цель мероприятия  практическая отработка навыков по оказанию помощи гражданам, попавшим в ДТП а также повышение и совершенствование уровня профессиональной подготовки сотрудников МЧС при проведении аварийно-спасательных работ по ликвидации последствий дорожно-транспортных проишествий сокращение временных показателей реагирования. | Основная цель мероприятия — практическая отработка навыков по оказанию помощи гражданам, попавшим в ДТП, а также повышение и совершенствование уровня профессиональной подготовки сотрудников МЧС при проведении аварийно-спасательных работ по ликвидации последствий дорожно-транспортных происшествий, сокращение временных показателей реагирования |
+|  |  |
+
+## Metrics
+### Quality
+Below are automatic metrics for determining the correctness of the spell checkers. 
+We compare our solution with both open automatic spell checkers and the ChatGPT family of models on all four available datasets:
+- **RUSpellRU**: texts collected from ([LiveJournal](https://www.livejournal.com/media)), with manually corrected typos and errors;
+- **MultidomainGold**: examples from 7 text sources, including the open web, news, social media, reviews, subtitles, policy documents and literary works;
+- **MedSpellChecker**: texts with errors from medical anamnesis;
+- **GitHubTypoCorpusRu**: spelling errors and typos in commits from [GitHub](https://github.com);
+
+**RUSpellRU**
+| Model | Pr. (spell) | Rec. (spell) | F1 (spell) | Pr. (punc) | Rec. (punc) | F1 (punc) | Pr. (case) | Rec. (case) | F1 (case) |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | 
+| sage-fredt5-large | 57.3 | 68.0 | 62.2 | 86.7 | 46.1 | 60.2 | 92.1 | 67.8 | 78.1 |
+| sage-fredt5-large (ft) | 88.4 | 80.9 | 84.5 | 88.2 | 85.3 | 86.8 | 95.5 | 94.0 | 94.7 |
+| sage-ai-service | 90.3 | 86.3 | 88.2 | 90.3 | 86.6 | 88.4 | 95.2 | 95.9 | 95.6 |
+| gpt-3.5-turbo | 33.6 | 58.5 | 42.7 | 85.9 | 64.6 | 73.7 | 84.9 | 73.9 | 79.0 |
+| gpt-4 | 54.9 | 76.7 | 64.0 | 84.0 | 82.3 | 83.2 | 91.5 | 90.2 | 90.9 |
+
+
+**MultidomainGold**
+| Model | Pr. (spell) | Rec. (spell) | F1 (spell) | Pr. (punc) | Rec. (punc) | F1 (punc) | Pr. (case) | Rec. (case) | F1 (case) |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | 
+| sage-fredt5-large | 43.4 | 49.7 | 46.3 | 21.8 | 21.3 | 21.6 | 58.8 | 23.9 | 34.0 |
+| sage-fredt5-large (ft) | 80.3 | 75.1 | 77.6 | 69.0 | 66.5 | 67.7 | 78.6 | 80.0 | 79.3 |
+| sage-ai-service | 81.6 | 77.7 | 79.6 | 70.2 | 67.5 | 68.8 | 80.5 | 80.5 | 80.5 |
+| gpt-3.5-turbo | 18.8 | 48.1 | 27.1 | 42.0 | 31.8 | 36.2 | 47.1 | 51.3 | 49.1 |
+| gpt-4 | 25.4 | 68.0 | 37.0 | 57.8 | 54.3 | 56.0 | 54.0 | 67.5 | 60.0 |
+
+
+**MedSpellChecker**
+| Model | Pr. (spell) | Rec. (spell) | F1 (spell) | Pr. (punc) | Rec. (punc) | F1 (punc) | Pr. (case) | Rec. (case) | F1 (case) |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | 
+| sage-fredt5-large | 35.2 | 54.5 | 42.8 | 19.2 | 13.2 | 15.7 | 48.7 | 36.8 | 41.9 |
+| sage-fredt5-large (ft) | 72.5 | 72.2 | 72.3 | 74.6 | 66.4 | 70.3 | 79.3 | 85.1 | 82.1 |
+| sage-ai-service | 71.3 | 73.5 | 72.4 | 75.1 | 69.2 | 72.0 | 80.9 | 72.8 | 76.6|
+| gpt-3.5-turbo | 14.7 | 45.9 | 22.3 | 69.9 | 52.3 | 59.8 | 26.4 | 41.8 | 32.3 |
+| gpt-4 | 37.8 | 72.3 | 49.6 | 81.4 | 64.3 | 71.9 | 73.0 | 62.1 | 67.1 |
+
+
+**GitHubTypoCorpusRu**
+| Model | Pr. (spell) | Rec. (spell) | F1 (spell) | Pr. (punc) | Rec. (punc) | F1 (punc) | Pr. (case) | Rec. (case) | F1 (case) |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | 
+| sage-fredt5-large | 46.0 | 46.6 | 46.3 | 22.7 | 18.3 | 20.2 | 12.0 | 13.2 | 12.6 |
+| sage-fredt5-large (ft) | 67.5 | 53.2 | 59.5 | 48.5  | 38.0 | 42.6 | 37.3 | 50.0 | 42.7 |
+| sage-ai-service | 70.8 | 56.3 | 62.7 | 48.9 | 35.8 | 41.4 | 32.9 | 45.3 | 38.1|
+| gpt-3.5-turbo | 23.7 | 38.7 | 29.4 | 37.6 | 23.3 | 28.7 | 19.6 | 35.9 | 25.3 |
+| gpt-4 | 27.0 | 52.8 | 35.7 | 45.9 | 32.6 | 38.2 | 25.7 | 36.8 | 30.2 |
+
+## How to use
+```python
+from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
+tokenizer = AutoTokenizer.from_pretrained("ai-forever/sage-fredt5-large")
+model = AutoModelForSeq2SeqLM.from_pretrained("ai-forever/sage-fredt5-large")
+model.to("cuda:0")
+sentence = "И не чсно прохожим в этот день непогожйи почему я веселый такйо"
+text = "<LM>" + sentence
+with torch.inference_mode():
+    encodings = tokenizer(text, max_length=None, padding="longest", truncation=False, return_tensors="pt")
+    for k, v in encodings.items():
+        encodings[k] = v.to("cuda:0")
+    res = model.generate(
+        **encodings,
+        use_cache=True,
+        max_length = encodings["input_ids"].size(1) * 1.5
+    )
+    res = res.cpu().tolist()
+    res = tokenizer.batch_decode(res, skip_special_tokens=True)
+print(res)
+# ["И не ясно прохожим в этот день непогожий, почему я веселый такой."]
+```
+
+## Limitations
+- The model is intended to be fine-tuned on sets with natural errors for better performance. The realized model is a pre-train and pre-train task is different from the usual spell checking in terms of density of the noise in a corpus and its origin;
+- Complex formatting may cause some trouble in output generation.
+
+## Resources
+- [SAGE library](https://github.com/ai-forever/sage), GitHub
+- [sage-fredt5-large](https://huggingface.co/ai-forever/sage-fredt5-large), HuggingFace
+- [sage-fredt5-distilled-95m](https://huggingface.co/ai-forever/sage-fredt5-distilled-95m), HuggingFace
+- [sage-m2m100-1.2B](https://huggingface.co/ai-forever/sage-m2m100-1.2B), HuggingFace
+- [sage-mt5-large](https://huggingface.co/ai-forever/sage-mt5-large), HuggingFace
+
+## License
+Model [FRED-T5-large](https://huggingface.co/ai-forever/FRED-T5-large), on the basis of which our solution is made, and its source code are supplied under the MIT license. 
+Our solution comes with MIT license also.
+
+## Specifications
+- File size: 3.3 Gb;
+- Framework: pytorch
+- Version: v1.0
+- Developer: SberDevices, AGI NLP
+
+## Contacts
+nikita.martynov.98@list.ru