set back-translation as main

README.md

@@ -13,8 +13,8 @@ library_name: fairseq
 
 ## Model description
 
-This model was trained from scratch using the [Fairseq toolkit](https://fairseq.readthedocs.io/en/latest/) on a combination of Catalan-German datasets, totalling 100.000.000 sentence pairs. 
-6.258.272 sentence pairs were parallel data collected from the web while the remaining 93.741.728 sentence pairs were parallel synthetic data created using the ES-CA translator of [PlanTL](https://huggingface.co/PlanTL-GOB-ES/mt-plantl-es-ca). The model was evaluated on the Flores and NTREX evaluation datasets. 
+This model was trained from scratch using the [Fairseq toolkit](https://fairseq.readthedocs.io/en/latest/) on a combination of Catalan-German datasets, 
+which after filtering and cleaning comprised 6.258.272 sentence pairs. The model was evaluated on the Flores and NTREX evaluation datasets. 
 
 ## Intended uses and limitations
 
@@ -36,7 +36,7 @@ import pyonmttok
 from huggingface_hub import snapshot_download
 model_dir = snapshot_download(repo_id="projecte-aina/aina-translator-ca-de", revision="main")
 
-tokenizer=pyonmttok.Tokenizer(mode="none", sp_model_path = model_dir + "/spm.50k.model")
+tokenizer=pyonmttok.Tokenizer(mode="none", sp_model_path = model_dir + "/spm.model")
 tokenized=tokenizer.tokenize("Benvingut al projecte Aina!")
 
 translator = ctranslate2.Translator(model_dir)
@@ -52,7 +52,7 @@ However, we are well aware that our models may be biased. We intend to conduct r
 
 ### Training data
 
-The Catalan-German data collected from the web was a combination of the following datasets:
+The model was trained on a combination of the following datasets:
 
 | Dataset       	| Sentences  	| Sentences after Cleaning|
 |-------------------|----------------|-------------------|
@@ -60,27 +60,19 @@ The Catalan-German data collected from the web was a combination of the followin
 | WikiMatrix  	| 180.322 	| 125.811 	|
 | GNOME	| 12.333|	1.241|
 | KDE4    	| 165.439   	|  105.098 	|
+| QED           	| 63.041 	|   49.181 	|
+| TED2020 v1    	| 46.680 	| 38.428 |
 | OpenSubtitles	| 303.329	| 171.376	|
 | GlobalVoices| 4.636 	|	3.578|
 | Tatoeba | 732 | 655 |
 | Books | 4.445 | 2049 |
 | Europarl | 1.734.643 | 1.734.643 |
 | Tilde | 3.434.091 | 3.434.091 |
+| **Total**     	| **7.427.843** | **6.258.272** |
 
 All corpora except Europarl and Tilde were collected from [Opus](https://opus.nlpl.eu/). 
-The Europarl and Tilde corpora are synthetic parallel corpora created from the original Spanish-German corpora by [SoftCatalà](https://github.com/Softcatala).
+The Europarl and Tilde corpora are synthetic parallel corpora created from the original Spanish-Catalan corpora by [SoftCatalà](https://github.com/Softcatala).
 
-The 93.741.728 sentence pairs of synthetic parallel data were created from the following Spanish-German datasets:
-
-| Dataset       	| Sentences before cleaning	|
-|-------------------|----------------|
-|globalvoices_es-de_20230901	|    70.097 |
-|multiparacrawl_es-de_20230901  | 56.873.541 |
-|dgt_es-de_20240129           |   4.899.734  | 
-|eubookshop_es-de_20240129    |   4.750.170  |
-|nllb_es-de_20240129          |   112.444.838 |
-|opensubtitles_es-de_20240129 |   18.951.214  |
-| **Total**                   | **197.989.594** |
 
 ### Training procedure
 
@@ -88,7 +80,8 @@ The 93.741.728 sentence pairs of synthetic parallel data were created from the f
 
  All datasets are deduplicated and filtered to remove any sentence pairs with a cosine similarity of less than 0.75. 
  This is done using sentence embeddings calculated using [LaBSE](https://huggingface.co/sentence-transformers/LaBSE). 
- The filtered datasets are then concatenated and before training the punctuation is normalized using a modified version of the join-single-file.py script from [SoftCatalà](https://github.com/Softcatala/nmt-models/blob/master/data-processing-tools/join-single-file.py)
+ The filtered datasets are then concatenated to form a final corpus of 6.159.631 and before training the punctuation is normalized 
+ using a modified version of the join-single-file.py script from [SoftCatalà](https://github.com/Softcatala/nmt-models/blob/master/data-processing-tools/join-single-file.py)
 
 
 #### Tokenization
@@ -137,10 +130,10 @@ and [Google Translate](https://translate.google.es/?hl=es):
 
 | Test set         	| SoftCatalà | Google Translate | aina-translator-ca-de |
 |----------------------|------------|------------------|---------------|
-| Flores 101 dev   	| 26,2     	| **34,8**     	| 34,1     	|
-| Flores 101 devtest   |26,3   	| **34,0**     	| 33,3     	|
-| NTREX | 21,7 | **28,8** | 27,8 |
-| Average          	| 24,7   	| **32,5**     	| 31,7      	|
+| Flores 101 dev   	| 26,2     	| **34,8**     	| 27,5     	|
+| Flores 101 devtest   |26,3   	| **34,0**     	| 26,9     	|
+| NTREX | 21,7 | **28,8** | 20,4 |
+| Average          	| 24,7   	| **32,5**     	| 24,9      	|
 
 ## Additional information
 

config.json

@@ -5,6 +5,5 @@
   "decoder_start_token": "</s>",
   "eos_token": "</s>",
   "layer_norm_epsilon": null,
-  "multi_query_attention": false,
   "unk_token": "<unk>"
 }

model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:38e98a6e9dd5e6c00cd94647c0695666d78c070bfb1c73ab70a36f7c6557a9e6
-size 1860811612
+oid sha256:e55f9e37a616e6d7cf7cc6111920e95133be662bbe4792cc6131f2df4ec25788
+size 1860745998

spm.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ec1143f19e1763ed954c4060f440ade847163d8267ec9e131b7c829080df8eb7
-size 1182306
+oid sha256:8999682b24246eb4bac8e43e528a47fe5555a7101710f04f4d3780804a703a77
+size 1182213