v2.0 with teacher all-mpnet-base-v2, trained with longer paragraphs

README.md

@@ -35,10 +35,18 @@ widget:
 
 # KBLab/sentence-bert-swedish-cased
 
-This is a [sentence-transformers](https://www.SBERT.net) model: It maps Swedish sentences & paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search. This model is a bilingual Swedish-English model trained according to instructions in the paper [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](https://arxiv.org/pdf/2004.09813.pdf) and the [documentation](https://www.sbert.net/examples/training/multilingual/README.html) accompanying its companion python package. We have used the strongest available pretrained English Bi-Encoder ([paraphrase-mpnet-base-v2](https://www.sbert.net/docs/pretrained_models.html#sentence-embedding-models)) as a teacher model, and the pretrained Swedish [KB-BERT](https://huggingface.co/KB/bert-base-swedish-cased) as the student model. 
+This is a [sentence-transformers](https://www.SBERT.net) model: It maps Swedish sentences & paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search. This model is a bilingual Swedish-English model trained according to instructions in the paper [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](https://arxiv.org/pdf/2004.09813.pdf) and the [documentation](https://www.sbert.net/examples/training/multilingual/README.html) accompanying its companion python package. We have used the strongest available pretrained English Bi-Encoder ([all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2)) as a teacher model, and the pretrained Swedish [KB-BERT](https://huggingface.co/KB/bert-base-swedish-cased) as the student model. 
 
 A more detailed description of the model can be found in an article we published on the [KBLab blog](https://kb-labb.github.io/posts/2021-08-23-a-swedish-sentence-transformer/). 
 
+**Update**: We have released updated versions of the model since the initial release. The original model described in the blog post is **v1.0**. The current version is **v2.0**. The newer versions are trained on longer paragraphs, and have a longer max sequence length. **v2.0** is trained with a stronger teacher model and is the current default.
+
+| Model version | Teacher Model | Max Sequence Length |
+|---------------|---------|----------|
+| v1.0          | [paraphrase-mpnet-base-v2](https://huggingface.co/sentence-transformers/paraphrase-mpnet-base-v2)  | 256   |
+| v1.1          | [paraphrase-mpnet-base-v2](https://huggingface.co/sentence-transformers/paraphrase-mpnet-base-v2)  | 384   |
+| v2.0          | [all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2)  | 384 |
+
 <!--- Describe your model here -->
 
 ## Usage (Sentence-Transformers)
@@ -81,6 +89,7 @@ def mean_pooling(model_output, attention_mask):
 sentences = ['Det här är en exempelmening', 'Varje exempel blir konverterad']
 
 # Load model from HuggingFace Hub
+# To load an older version, e.g. v1.0, add the argument revision="v1.0" 
 tokenizer = AutoTokenizer.from_pretrained('KBLab/sentence-bert-swedish-cased')
 model = AutoModel.from_pretrained('KBLab/sentence-bert-swedish-cased')
 
@@ -98,13 +107,19 @@ print("Sentence embeddings:")
 print(sentence_embeddings)
 ```
 
-
+To load an older model specify the version tag with the `revision` arg: `AutoTokenizer.from_pretrained('KBLab/sentence-bert-swedish-cased', revision="v1.0")`.
 
 ## Evaluation Results
 
 <!--- Describe how your model was evaluated -->
 
-The model was primarily evaluated on [SweParaphrase v1.0](https://spraakbanken.gu.se/en/resources/sweparaphrase). This test set is part of [SuperLim](https://spraakbanken.gu.se/en/resources/superlim) -- a Swedish evaluation suite for natural langage understanding tasks.  We calculated Pearson and Spearman correlation between predicted model similarity scores and the human similarity score labels. The model achieved a Pearson correlation coefficient of **0.918** and a Spearman's rank correlation coefficient of **0.911**.
+The model was evaluated on [SweParaphrase v1.0](https://spraakbanken.gu.se/en/resources/sweparaphrase) and **SweParaphrase v2.0**. This test set is part of [SuperLim](https://spraakbanken.gu.se/en/resources/superlim) -- a Swedish evaluation suite for natural langage understanding tasks.  We calculated Pearson and Spearman correlation between predicted model similarity scores and the human similarity score labels. Results from **SweParaphrase v1.0** are displayed below. 
+
+| Model version | Pearson | Spearman |
+|---------------|---------|----------|
+| v1.0          | 0.9183  | 0.9114   |
+| v1.1          | 0.9183  | 0.9114   |
+| v2.0          | **0.9283**  | **0.9130**   |
 
 The following code snippet can be used to reproduce the above results:
 
@@ -149,13 +164,46 @@ print(df[["score", "model_score"]].corr(method="spearman"))
 print(df[["score", "model_score"]].corr(method="pearson"))
 ```
 
-Examples how to evaluate the model on other test sets of the SuperLim suites can be found on the following links: [evaluate_faq.py](https://github.com/kb-labb/swedish-sbert/blob/main/evaluate_faq.py) (Swedish FAQ), [evaluate_swesat.py](https://github.com/kb-labb/swedish-sbert/blob/main/evaluate_swesat.py) (SweSAT synonyms), [evaluate_supersim.py](https://github.com/kb-labb/swedish-sbert/blob/main/evaluate_supersim.py) (SuperSim).
+### Sweparaphrase v2.0
+
+In general, **v1.1** correlates the most with human assessment of text similarity on SweParaphrase v2.0. Below, we present zero-shot evaluation results on all data splits. They display the model's performance out of the box, without any fine-tuning.
+
+| Model version | Data split | Pearson    | Spearman   |
+|---------------|------------|------------|------------|
+| v1.0          | train      | 0.8355     | 0.8256     |
+| v1.1          | train      | **0.8383** | **0.8302** |
+| v2.0          | train      | 0.8209     | 0.8059     |
+| v1.0          | dev        | 0.8682     | 0.8774     |
+| v1.1          | dev        | **0.8739** | **0.8833** |
+| v2.0          | dev        | 0.8638     | 0.8668     |
+| v1.0          | test       | 0.8356     | 0.8476     |
+| v1.1          | test       | **0.8393** | **0.8550** |
+| v2.0          | test       | 0.8232     | 0.8213     |
+
+### SweFAQ v2.0
+
+When it comes to retrieval tasks, **v2.0** performs the best by quite a substantial margin. It is better at matching the correct answer to a question compared to v1.1 and v1.0.
+
+| Model version | Data split | Accuracy   |
+|---------------|------------|------------|
+| v1.0          | train      | 0.5262     |
+| v1.1          | train      | 0.6236     |
+| v2.0          | train      | **0.7106** |
+| v1.0          | dev        | 0.4636     |
+| v1.1          | dev        | 0.5818     |
+| v2.0          | dev        | **0.6727** |
+| v1.0          | test       | 0.4495     |
+| v1.1          | test       | 0.5229     |
+| v2.0          | test       | **0.5871** |
+
+
+Examples how to evaluate the models on some of the test sets of the SuperLim suites can be found on the following links: [evaluate_faq.py](https://github.com/kb-labb/swedish-sbert/blob/main/evaluate_faq.py) (Swedish FAQ), [evaluate_swesat.py](https://github.com/kb-labb/swedish-sbert/blob/main/evaluate_swesat.py) (SweSAT synonyms), [evaluate_supersim.py](https://github.com/kb-labb/swedish-sbert/blob/main/evaluate_supersim.py) (SuperSim).
 
 ## Training
 
-An article with more details on data and the model can be found on the [KBLab blog](https://kb-labb.github.io/posts/2021-08-23-a-swedish-sentence-transformer/). 
+An article with more details on data and v1.0 of the model can be found on the [KBLab blog](https://kb-labb.github.io/posts/2021-08-23-a-swedish-sentence-transformer/). 
 
-Around 14.6 million sentences from English-Swedish parallel corpuses were used to train the model. Data was sourced from the [Open Parallel Corpus](https://opus.nlpl.eu/) (OPUS) and downloaded via the python package [opustools](https://pypi.org/project/opustools/). Datasets used were: JW300, Europarl, EUbookshop, EMEA, TED2020, Tatoeba and OpenSubtitles. 
+Around 14.6 million sentences from English-Swedish parallel corpuses were used to train the model. Data was sourced from the [Open Parallel Corpus](https://opus.nlpl.eu/) (OPUS) and downloaded via the python package [opustools](https://pypi.org/project/opustools/). Datasets used were: JW300, Europarl, DGT-TM, EMEA, ELITR-ECA, TED2020, Tatoeba and OpenSubtitles. 
 
 The model was trained with the parameters:
 

config.json

@@ -1,5 +1,5 @@
 {
-  "_name_or_path": "output/original-en-sv-2022-12-20_16-52-40/",
+  "_name_or_path": "output/no-normalize-en-sv-2022-12-26_18-39-42/",
   "architectures": [
     "BertModel"
   ],

eval/mse_evaluation_TED2020-en-sv-dev.tsv.gz_results.csv

@@ -1,124 +1,117 @@
 epoch,steps,MSE
-0,1000,0.34974124282598495
-0,2000,0.34921395126730204
-0,3000,0.354314292781055
-0,4000,0.3488133195787668
-0,5000,0.34792646765708923
-0,6000,0.36894388031214476
-0,7000,0.35022159572690725
-0,8000,0.3502144478261471
-0,9000,0.3629593178629875
-0,10000,0.34892880357801914
-0,11000,0.34935250878334045
-0,12000,0.3478859318420291
-0,13000,0.3487293142825365
-0,14000,0.34829590003937483
-0,15000,0.34732643980532885
-0,16000,0.34766194876283407
-0,17000,0.3484585788100958
-0,18000,0.34895502030849457
-0,19000,0.34811608493328094
-0,20000,0.34691705368459225
-0,21000,0.3468364477157593
-0,22000,0.34657707437872887
-0,23000,0.34666003193706274
-0,24000,0.3460049396380782
-0,25000,0.34664138220250607
-0,26000,0.34586922265589237
-0,27000,0.3461060579866171
-0,28000,0.3462697612121701
-0,29000,0.34733759239315987
-0,30000,0.34663842525333166
-0,31000,0.34415228292346
-0,32000,0.34451447427272797
-0,33000,0.344709912315011
-0,34000,0.34585813991725445
-0,35000,0.3458136925473809
-0,36000,0.3447240451350808
-0,37000,0.34575818572193384
-0,38000,0.3454188583418727
-0,39000,0.3460196079686284
-0,40000,0.3453935729339719
-0,41000,0.3460376523435116
-0,42000,0.3455940168350935
-0,43000,0.3460632869973779
-0,44000,0.34571660216897726
-0,45000,0.3458129009231925
-0,46000,0.34544935915619135
-0,47000,0.3456356469541788
-0,48000,0.3457914339378476
-0,49000,0.34663709811866283
-0,50000,0.34600410144776106
-0,51000,0.34615208860486746
-0,52000,0.3460976528003812
-0,53000,0.3433866426348686
-0,54000,0.3440452506765723
-0,55000,0.34487603697925806
-0,56000,0.3436979604884982
-0,57000,0.3439406165853143
-0,58000,0.3453923622146249
-0,59000,0.34493771381676197
-0,60000,0.3449903568252921
-0,61000,0.3443687688559294
-0,62000,0.3443889319896698
-0,63000,0.34302377607673407
-0,64000,0.34372734371572733
-0,65000,0.344184716232121
-0,66000,0.3440574277192354
-0,67000,0.34405707847326994
-0,68000,0.34330685157328844
-0,69000,0.3439192660152912
-0,70000,0.34411929082125425
-0,71000,0.3435770748183131
-0,72000,0.34372536465525627
-0,73000,0.34490013495087624
-0,74000,0.34326689783483744
-0,75000,0.34309893380850554
-0,76000,0.34250954631716013
-0,77000,0.3431871999055147
-0,78000,0.3428747411817312
-0,79000,0.3436204046010971
-0,80000,0.3433616366237402
-0,81000,0.34336880780756474
-0,82000,0.3433717181906104
-0,83000,0.3438536310568452
-0,84000,0.34293336793780327
-0,85000,0.3442385932430625
-0,86000,0.3430124372243881
-0,87000,0.3432458033785224
-0,88000,0.3439270658418536
-0,89000,0.34296554513275623
-0,90000,0.3425579285249114
-0,91000,0.343247945420444
-0,92000,0.3435784252360463
-0,93000,0.34204861149191856
-0,94000,0.3420257940888405
-0,95000,0.3410392440855503
-0,96000,0.3406720468774438
-0,97000,0.3412422724068165
-0,98000,0.34140332136303186
-0,99000,0.3417556406930089
-0,100000,0.34126145765185356
-0,101000,0.34189000725746155
-0,102000,0.34180444199591875
-0,103000,0.3405329305678606
-0,104000,0.34238805528730154
-0,105000,0.34229636657983065
-0,106000,0.34293823409825563
-0,107000,0.3425850532948971
-0,108000,0.3417897270992398
-0,109000,0.34169263672083616
-0,110000,0.3422217909246683
-0,111000,0.34196663182228804
-0,112000,0.3404703689739108
-0,113000,0.3403896000236273
-0,114000,0.3410223638638854
-0,115000,0.3409507218748331
-0,116000,0.34049374517053366
-0,117000,0.3408001968637109
-0,118000,0.3406590083613992
-0,119000,0.34159310162067413
-0,120000,0.3412698395550251
-0,121000,0.3411141689866781
-0,122000,0.34038866870105267
-0,123000,0.34109046682715416
+0,1000,0.3305374411866069
+0,2000,0.3307490376755595
+0,3000,0.33029515761882067
+0,4000,0.3363496856763959
+0,5000,0.33181568142026663
+0,6000,0.33005450386554
+0,7000,0.3400696674361825
+0,8000,0.3337323432788253
+0,9000,0.3324012039229274
+0,10000,0.3307985607534647
+0,11000,0.33261950593441725
+0,12000,0.3310671541839838
+0,13000,0.3318543080240488
+0,14000,0.33143835607916117
+0,15000,0.33053881488740444
+0,16000,0.33193877898156643
+0,17000,0.33089425414800644
+0,18000,0.3306396072730422
+0,19000,0.3306496189907193
+0,20000,0.3303301753476262
+0,21000,0.33140445593744516
+0,22000,0.33020293340086937
+0,23000,0.3308977698907256
+0,24000,0.3299850504845381
+0,25000,0.3310497384518385
+0,26000,0.33014328218996525
+0,27000,0.3300098003819585
+0,28000,0.3302561352029443
+0,29000,0.33076435793191195
+0,30000,0.33028211910277605
+0,31000,0.33056996762752533
+0,32000,0.3295192960649729
+0,33000,0.32972143962979317
+0,34000,0.3295718925073743
+0,35000,0.3292877459898591
+0,36000,0.32971047330647707
+0,37000,0.3298777621239424
+0,38000,0.32897726632654667
+0,39000,0.32850170973688364
+0,40000,0.32882699742913246
+0,41000,0.3295514499768615
+0,42000,0.3299229545518756
+0,43000,0.3288676030933857
+0,44000,0.3289586631581187
+0,45000,0.3293627640232444
+0,46000,0.3293595276772976
+0,47000,0.32900278456509113
+0,48000,0.3292385721579194
+0,49000,0.3301975317299366
+0,50000,0.3289953572675586
+0,51000,0.3290593158453703
+0,52000,0.32869731076061726
+0,53000,0.32881496008485556
+0,54000,0.32918842043727636
+0,55000,0.32880869694054127
+0,56000,0.3290021326392889
+0,57000,0.3285171231254935
+0,58000,0.3282122313976288
+0,59000,0.32952832989394665
+0,60000,0.3284840611740947
+0,61000,0.32849146518856287
+0,62000,0.3292925423011184
+0,63000,0.32826855313032866
+0,64000,0.32868378330022097
+0,65000,0.3289590124040842
+0,66000,0.32914113253355026
+0,67000,0.32902946695685387
+0,68000,0.327494740486145
+0,69000,0.32870552968233824
+0,70000,0.32899840734899044
+0,71000,0.3288975451141596
+0,72000,0.32872746232897043
+0,73000,0.3284345380961895
+0,74000,0.32932700123637915
+0,75000,0.3289450891315937
+0,76000,0.32835565507411957
+0,77000,0.3284606384113431
+0,78000,0.3285201732069254
+0,79000,0.3282968420535326
+0,80000,0.3280844073742628
+0,81000,0.3282552817836404
+0,82000,0.32890671864151955
+0,83000,0.3278125077486038
+0,84000,0.32847451511770487
+0,85000,0.3284695325419307
+0,86000,0.3288332372903824
+0,87000,0.32888082787394524
+0,88000,0.32766179647296667
+0,89000,0.3283764934167266
+0,90000,0.32793665304780006
+0,91000,0.32725471537560225
+0,92000,0.3277936251834035
+0,93000,0.3274726215749979
+0,94000,0.32755015417933464
+0,95000,0.3280520439147949
+0,96000,0.3282654797658324
+0,97000,0.32788922544568777
+0,98000,0.32690519001334906
+0,99000,0.32813241705298424
+0,100000,0.3279812401160598
+0,101000,0.32842066138982773
+0,102000,0.3276278730481863
+0,103000,0.32748677767813206
+0,104000,0.3282419638708234
+0,105000,0.3277064301073551
+0,106000,0.32805497758090496
+0,107000,0.3275437746196985
+0,108000,0.32795085571706295
+0,109000,0.32730509992688894
+0,110000,0.32666658516973257
+0,111000,0.3273332491517067
+0,112000,0.32759017776697874
+0,113000,0.32762193586677313
+0,114000,0.32658560667186975
+0,115000,0.3273524809628725
+0,116000,0.3271533874794841

eval/mse_evaluation_Tatoeba-eng-swe-dev.tsv.gz_results.csv

@@ -1,124 +1,117 @@
 epoch,steps,MSE
-0,1000,0.2744395984336734
-0,2000,0.2732344903051853
-0,3000,0.27526929043233395
-0,4000,0.2727509709075093
-0,5000,0.2726479200646281
-0,6000,0.2773450221866369
-0,7000,0.2734815236181021
-0,8000,0.27323723770678043
-0,9000,0.27514498215168715
-0,10000,0.27173745911568403
-0,11000,0.271493848413229
-0,12000,0.2710314467549324
-0,13000,0.2713753143325448
-0,14000,0.27073214296251535
-0,15000,0.2711733803153038
-0,16000,0.27131817769259214
-0,17000,0.2712530782446265
-0,18000,0.2713564317673445
-0,19000,0.27065700851380825
-0,20000,0.2708346117287874
-0,21000,0.2701618243008852
-0,22000,0.2708142623305321
-0,23000,0.27058098930865526
-0,24000,0.26987355668097734
-0,25000,0.2702167257666588
-0,26000,0.27005996089428663
-0,27000,0.2704422688111663
-0,28000,0.27046180330216885
-0,29000,0.26996054220944643
-0,30000,0.2703654346987605
-0,31000,0.2706443890929222
-0,32000,0.2700826618820429
-0,33000,0.2698082709684968
-0,34000,0.27026834432035685
-0,35000,0.2701908349990845
-0,36000,0.26949571911245584
-0,37000,0.2693337853997946
-0,38000,0.26938540395349264
-0,39000,0.26982848066836596
-0,40000,0.2692650770768523
-0,41000,0.269907433539629
-0,42000,0.2692088717594743
-0,43000,0.26961518451571465
-0,44000,0.2693982794880867
-0,45000,0.26962801348418
-0,46000,0.2698043594136834
-0,47000,0.2695266157388687
-0,48000,0.26929995510727167
-0,49000,0.26941525284200907
-0,50000,0.2691812813282013
-0,51000,0.26892221067100763
-0,52000,0.2696847543120384
-0,53000,0.268734572455287
-0,54000,0.26862353552132845
-0,55000,0.26856022886931896
-0,56000,0.2691086381673813
-0,57000,0.26887678541243076
-0,58000,0.2694257069379091
-0,59000,0.2690558088943362
-0,60000,0.26923459954559803
-0,61000,0.2686571329832077
-0,62000,0.2689000219106674
-0,63000,0.2692748326808214
-0,64000,0.2686175983399153
-0,65000,0.2685254439711571
-0,66000,0.26889864820986986
-0,67000,0.26836462784558535
-0,68000,0.2682679798454046
-0,69000,0.26815461460500956
-0,70000,0.2683368744328618
-0,71000,0.2682250924408436
-0,72000,0.2687834436073899
-0,73000,0.26929827872663736
-0,74000,0.2690488239750266
-0,75000,0.2679044147953391
-0,76000,0.2682675374671817
-0,77000,0.26816066820174456
-0,78000,0.2677485812455416
-0,79000,0.2684414852410555
-0,80000,0.26813503354787827
-0,81000,0.2680476289242506
-0,82000,0.26812972500920296
-0,83000,0.2681598300114274
-0,84000,0.26870507281273603
-0,85000,0.2689675893634558
-0,86000,0.2684480743482709
-0,87000,0.2686573890969157
-0,88000,0.2692556008696556
-0,89000,0.2688751555979252
-0,90000,0.2683075377717614
-0,91000,0.26865273248404264
-0,92000,0.26835002936422825
-0,93000,0.2683911705389619
-0,94000,0.2683736849576235
-0,95000,0.2682714257389307
-0,96000,0.2680947305634618
-0,97000,0.2681643469259143
-0,98000,0.26855955366045237
-0,99000,0.2683456987142563
-0,100000,0.26787673123180866
-0,101000,0.26841198559850454
-0,102000,0.26806306559592485
-0,103000,0.2671808935701847
-0,104000,0.267979153431952
-0,105000,0.2673288341611624
-0,106000,0.26792525313794613
-0,107000,0.2680175472050905
-0,108000,0.267645507119596
-0,109000,0.2677440643310547
-0,110000,0.2679037628695369
-0,111000,0.2675337716937065
-0,112000,0.26777954772114754
-0,113000,0.2673337934538722
-0,114000,0.2673886017873883
-0,115000,0.26709178928285837
-0,116000,0.2673146314918995
-0,117000,0.26732038240879774
-0,118000,0.2673294395208359
-0,119000,0.26773312129080296
-0,120000,0.26698445435613394
-0,121000,0.2667626366019249
-0,122000,0.2666329964995384
-0,123000,0.2665518783032894
+0,1000,0.24785797577351332
+0,2000,0.2478782320395112
+0,3000,0.24820237886160612
+0,4000,0.25052379351109266
+0,5000,0.2484829630702734
+0,6000,0.24821306578814983
+0,7000,0.25101194623857737
+0,8000,0.24918639101088047
+0,9000,0.24825208820402622
+0,10000,0.24815460201352835
+0,11000,0.24851374328136444
+0,12000,0.24818778038024902
+0,13000,0.247632572427392
+0,14000,0.24752533063292503
+0,15000,0.24750018492341042
+0,16000,0.24753324687480927
+0,17000,0.24697233457118273
+0,18000,0.24762307293713093
+0,19000,0.24796028155833483
+0,20000,0.24713336024433374
+0,21000,0.2480252180248499
+0,22000,0.24814684875309467
+0,23000,0.24748872965574265
+0,24000,0.24726693518459797
+0,25000,0.24767774157226086
+0,26000,0.2473350614309311
+0,27000,0.24677005130797625
+0,28000,0.24716746993362904
+0,29000,0.24732353631407022
+0,30000,0.2474617213010788
+0,31000,0.24711331352591515
+0,32000,0.24705547839403152
+0,33000,0.24687713012099266
+0,34000,0.24697906337678432
+0,35000,0.24673829320818186
+0,36000,0.24703482631593943
+0,37000,0.24725922849029303
+0,38000,0.24706728290766478
+0,39000,0.2470718463882804
+0,40000,0.24706239346414804
+0,41000,0.247084628790617
+0,42000,0.24669873528182507
+0,43000,0.2467589918524027
+0,44000,0.24676024913787842
+0,45000,0.24646525271236897
+0,46000,0.24657496251165867
+0,47000,0.245952932164073
+0,48000,0.24603260681033134
+0,49000,0.2463929122313857
+0,50000,0.24623936042189598
+0,51000,0.24639982730150223
+0,52000,0.2464748453348875
+0,53000,0.24611358530819416
+0,54000,0.24669363629072905
+0,55000,0.2464905148372054
+0,56000,0.24694388266652822
+0,57000,0.24669785052537918
+0,58000,0.24613626301288605
+0,59000,0.2463148208335042
+0,60000,0.24608178064227104
+0,61000,0.24618220049887896
+0,62000,0.24709079880267382
+0,63000,0.2463518874719739
+0,64000,0.24638932663947344
+0,65000,0.2465276513248682
+0,66000,0.24644010700285435
+0,67000,0.24675603490322828
+0,68000,0.2460342599079013
+0,69000,0.24680779315531254
+0,70000,0.24674353189766407
+0,71000,0.24644036311656237
+0,72000,0.24643116630613804
+0,73000,0.24601935874670744
+0,74000,0.24650872219353914
+0,75000,0.2464913995936513
+0,76000,0.24664695374667645
+0,77000,0.24642888456583023
+0,78000,0.24638038594275713
+0,79000,0.24592014960944653
+0,80000,0.24589370004832745
+0,81000,0.2457715105265379
+0,82000,0.24643635842949152
+0,83000,0.24539267178624868
+0,84000,0.24630383122712374
+0,85000,0.2461036667227745
+0,86000,0.24639442563056946
+0,87000,0.24640655610710382
+0,88000,0.2456542570143938
+0,89000,0.2460445510223508
+0,90000,0.24578433949500322
+0,91000,0.24577109143137932
+0,92000,0.24596715811640024
+0,93000,0.2458097180351615
+0,94000,0.24577626027166843
+0,95000,0.24602224584668875
+0,96000,0.24567588698118925
+0,97000,0.24607458617538214
+0,98000,0.24560708552598953
+0,99000,0.2458440838381648
+0,100000,0.24566156789660454
+0,101000,0.24600222241133451
+0,102000,0.24566147476434708
+0,103000,0.2461111405864358
+0,104000,0.2459018025547266
+0,105000,0.24577155709266663
+0,106000,0.2457206603139639
+0,107000,0.24565793573856354
+0,108000,0.24591167457401752
+0,109000,0.24543707258999348
+0,110000,0.24535527918487787
+0,111000,0.2456084592267871
+0,112000,0.24576487485319376
+0,113000,0.24569914676249027
+0,114000,0.24549257941544056
+0,115000,0.24552540853619576
+0,116000,0.2456445712596178

eval/translation_evaluation_TED2020-en-sv-dev.tsv.gz_results.csv

@@ -1,124 +1,117 @@
 epoch,steps,src2trg,trg2src
-0,1000,0.973,0.971
-0,2000,0.973,0.973
-0,3000,0.974,0.969
-0,4000,0.974,0.97
-0,5000,0.974,0.972
-0,6000,0.975,0.97
-0,7000,0.975,0.971
-0,8000,0.974,0.97
-0,9000,0.976,0.968
-0,10000,0.974,0.97
-0,11000,0.975,0.972
-0,12000,0.975,0.969
-0,13000,0.975,0.97
-0,14000,0.975,0.97
-0,15000,0.975,0.967
-0,16000,0.973,0.971
-0,17000,0.974,0.973
-0,18000,0.974,0.971
-0,19000,0.974,0.971
-0,20000,0.975,0.972
-0,21000,0.975,0.971
-0,22000,0.974,0.974
-0,23000,0.974,0.974
-0,24000,0.974,0.973
-0,25000,0.974,0.971
-0,26000,0.975,0.972
-0,27000,0.975,0.972
-0,28000,0.974,0.973
-0,29000,0.973,0.973
-0,30000,0.974,0.973
-0,31000,0.975,0.973
-0,32000,0.975,0.973
-0,33000,0.975,0.973
-0,34000,0.975,0.973
-0,35000,0.975,0.973
-0,36000,0.975,0.973
-0,37000,0.975,0.972
-0,38000,0.974,0.972
-0,39000,0.974,0.972
-0,40000,0.976,0.973
-0,41000,0.976,0.973
-0,42000,0.976,0.973
-0,43000,0.975,0.973
-0,44000,0.974,0.972
-0,45000,0.975,0.972
-0,46000,0.975,0.972
-0,47000,0.974,0.972
-0,48000,0.974,0.971
-0,49000,0.976,0.973
-0,50000,0.975,0.972
-0,51000,0.975,0.972
-0,52000,0.976,0.973
-0,53000,0.975,0.972
-0,54000,0.975,0.973
-0,55000,0.976,0.973
-0,56000,0.975,0.973
-0,57000,0.976,0.973
-0,58000,0.975,0.973
-0,59000,0.975,0.973
-0,60000,0.976,0.973
-0,61000,0.976,0.973
-0,62000,0.976,0.972
-0,63000,0.975,0.973
-0,64000,0.976,0.972
-0,65000,0.974,0.973
-0,66000,0.975,0.973
-0,67000,0.975,0.973
-0,68000,0.975,0.973
-0,69000,0.975,0.973
-0,70000,0.975,0.972
-0,71000,0.975,0.972
-0,72000,0.975,0.972
-0,73000,0.974,0.972
-0,74000,0.975,0.973
-0,75000,0.976,0.973
-0,76000,0.975,0.973
-0,77000,0.976,0.973
-0,78000,0.976,0.972
-0,79000,0.976,0.973
-0,80000,0.975,0.973
-0,81000,0.975,0.974
-0,82000,0.976,0.973
-0,83000,0.976,0.973
-0,84000,0.976,0.973
-0,85000,0.976,0.974
-0,86000,0.975,0.974
-0,87000,0.975,0.974
-0,88000,0.975,0.974
-0,89000,0.975,0.974
-0,90000,0.976,0.974
-0,91000,0.975,0.974
-0,92000,0.975,0.973
-0,93000,0.975,0.973
-0,94000,0.976,0.973
-0,95000,0.976,0.973
-0,96000,0.976,0.974
-0,97000,0.976,0.973
-0,98000,0.976,0.973
-0,99000,0.975,0.974
-0,100000,0.975,0.973
-0,101000,0.975,0.973
-0,102000,0.974,0.974
-0,103000,0.976,0.974
-0,104000,0.975,0.974
-0,105000,0.976,0.974
-0,106000,0.974,0.973
-0,107000,0.977,0.973
-0,108000,0.976,0.974
-0,109000,0.975,0.974
-0,110000,0.976,0.974
-0,111000,0.976,0.974
-0,112000,0.975,0.974
-0,113000,0.976,0.975
-0,114000,0.977,0.974
-0,115000,0.976,0.974
-0,116000,0.975,0.975
-0,117000,0.975,0.974
-0,118000,0.976,0.974
-0,119000,0.976,0.974
-0,120000,0.976,0.974
-0,121000,0.976,0.974
-0,122000,0.976,0.974
-0,123000,0.975,0.974
+0,1000,0.971,0.971
+0,2000,0.971,0.971
+0,3000,0.97,0.972
+0,4000,0.971,0.972
+0,5000,0.971,0.971
+0,6000,0.971,0.972
+0,7000,0.971,0.972
+0,8000,0.971,0.973
+0,9000,0.97,0.971
+0,10000,0.97,0.97
+0,11000,0.97,0.97
+0,12000,0.97,0.971
+0,13000,0.969,0.97
+0,14000,0.97,0.971
+0,15000,0.97,0.971
+0,16000,0.97,0.971
+0,17000,0.971,0.971
+0,18000,0.971,0.971
+0,19000,0.97,0.971
+0,20000,0.97,0.971
+0,21000,0.97,0.971
+0,22000,0.97,0.971
+0,23000,0.971,0.971
+0,24000,0.97,0.971
+0,25000,0.97,0.971
+0,26000,0.97,0.97
+0,27000,0.97,0.971
+0,28000,0.97,0.971
+0,29000,0.97,0.97
+0,30000,0.969,0.971
+0,31000,0.97,0.97
+0,32000,0.97,0.97
+0,33000,0.969,0.971
+0,34000,0.97,0.971
+0,35000,0.97,0.969
+0,36000,0.97,0.971
+0,37000,0.97,0.969
+0,38000,0.97,0.971
+0,39000,0.97,0.971
+0,40000,0.971,0.971
+0,41000,0.97,0.971
+0,42000,0.97,0.971
+0,43000,0.97,0.971
+0,44000,0.971,0.971
+0,45000,0.969,0.971
+0,46000,0.969,0.972
+0,47000,0.971,0.971
+0,48000,0.971,0.971
+0,49000,0.971,0.97
+0,50000,0.97,0.971
+0,51000,0.972,0.97
+0,52000,0.97,0.971
+0,53000,0.97,0.971
+0,54000,0.97,0.971
+0,55000,0.97,0.971
+0,56000,0.97,0.971
+0,57000,0.97,0.971
+0,58000,0.969,0.971
+0,59000,0.972,0.971
+0,60000,0.971,0.97
+0,61000,0.971,0.971
+0,62000,0.971,0.971
+0,63000,0.971,0.97
+0,64000,0.971,0.971
+0,65000,0.971,0.971
+0,66000,0.971,0.971
+0,67000,0.971,0.971
+0,68000,0.97,0.971
+0,69000,0.971,0.971
+0,70000,0.971,0.971
+0,71000,0.971,0.97
+0,72000,0.971,0.971
+0,73000,0.971,0.971
+0,74000,0.97,0.971
+0,75000,0.971,0.971
+0,76000,0.971,0.97
+0,77000,0.971,0.971
+0,78000,0.971,0.971
+0,79000,0.971,0.97
+0,80000,0.97,0.971
+0,81000,0.97,0.971
+0,82000,0.971,0.97
+0,83000,0.971,0.97
+0,84000,0.971,0.97
+0,85000,0.97,0.97
+0,86000,0.971,0.971
+0,87000,0.971,0.971
+0,88000,0.971,0.971
+0,89000,0.971,0.97
+0,90000,0.97,0.97
+0,91000,0.971,0.971
+0,92000,0.97,0.97
+0,93000,0.97,0.97
+0,94000,0.97,0.971
+0,95000,0.97,0.97
+0,96000,0.97,0.97
+0,97000,0.97,0.971
+0,98000,0.97,0.971
+0,99000,0.971,0.971
+0,100000,0.97,0.971
+0,101000,0.971,0.971
+0,102000,0.97,0.971
+0,103000,0.97,0.97
+0,104000,0.97,0.971
+0,105000,0.97,0.971
+0,106000,0.971,0.97
+0,107000,0.97,0.97
+0,108000,0.971,0.971
+0,109000,0.971,0.97
+0,110000,0.971,0.97
+0,111000,0.971,0.971
+0,112000,0.971,0.971
+0,113000,0.971,0.971
+0,114000,0.971,0.97
+0,115000,0.97,0.969
+0,116000,0.97,0.97

eval/translation_evaluation_Tatoeba-eng-swe-dev.tsv.gz_results.csv

@@ -1,124 +1,117 @@
 epoch,steps,src2trg,trg2src
-0,1000,0.97,0.97
-0,2000,0.968,0.97
-0,3000,0.967,0.97
-0,4000,0.967,0.969
-0,5000,0.967,0.969
-0,6000,0.969,0.969
-0,7000,0.967,0.969
-0,8000,0.967,0.971
-0,9000,0.966,0.968
-0,10000,0.966,0.971
-0,11000,0.965,0.97
-0,12000,0.965,0.97
-0,13000,0.967,0.97
-0,14000,0.969,0.97
-0,15000,0.969,0.97
-0,16000,0.967,0.97
-0,17000,0.967,0.97
-0,18000,0.967,0.97
-0,19000,0.965,0.97
-0,20000,0.966,0.969
-0,21000,0.967,0.969
-0,22000,0.967,0.97
-0,23000,0.966,0.969
-0,24000,0.966,0.97
-0,25000,0.965,0.971
-0,26000,0.965,0.97
-0,27000,0.966,0.97
-0,28000,0.967,0.971
-0,29000,0.968,0.97
-0,30000,0.969,0.97
-0,31000,0.969,0.968
-0,32000,0.967,0.969
-0,33000,0.967,0.969
-0,34000,0.967,0.969
-0,35000,0.968,0.97
-0,36000,0.968,0.97
-0,37000,0.967,0.968
-0,38000,0.968,0.97
-0,39000,0.967,0.968
-0,40000,0.966,0.968
-0,41000,0.966,0.969
-0,42000,0.967,0.97
-0,43000,0.967,0.97
-0,44000,0.967,0.97
-0,45000,0.967,0.969
-0,46000,0.965,0.968
-0,47000,0.966,0.969
-0,48000,0.965,0.969
-0,49000,0.965,0.967
-0,50000,0.966,0.97
-0,51000,0.967,0.97
-0,52000,0.965,0.97
-0,53000,0.967,0.969
-0,54000,0.967,0.969
-0,55000,0.965,0.97
-0,56000,0.967,0.969
-0,57000,0.968,0.97
-0,58000,0.966,0.97
-0,59000,0.967,0.97
-0,60000,0.97,0.969
-0,61000,0.967,0.97
-0,62000,0.967,0.969
-0,63000,0.967,0.969
-0,64000,0.966,0.969
-0,65000,0.967,0.97
-0,66000,0.968,0.97
-0,67000,0.965,0.969
-0,68000,0.966,0.97
-0,69000,0.967,0.97
-0,70000,0.966,0.97
-0,71000,0.964,0.968
-0,72000,0.965,0.969
-0,73000,0.968,0.97
-0,74000,0.967,0.97
-0,75000,0.966,0.97
-0,76000,0.967,0.969
-0,77000,0.967,0.969
-0,78000,0.966,0.969
-0,79000,0.966,0.97
-0,80000,0.966,0.969
-0,81000,0.966,0.969
-0,82000,0.966,0.968
-0,83000,0.965,0.969
-0,84000,0.967,0.969
-0,85000,0.966,0.969
-0,86000,0.966,0.969
-0,87000,0.967,0.968
-0,88000,0.966,0.968
-0,89000,0.967,0.969
-0,90000,0.968,0.967
-0,91000,0.967,0.968
-0,92000,0.966,0.969
-0,93000,0.967,0.968
-0,94000,0.968,0.969
-0,95000,0.967,0.969
-0,96000,0.968,0.969
-0,97000,0.967,0.97
-0,98000,0.969,0.968
-0,99000,0.967,0.969
-0,100000,0.966,0.97
-0,101000,0.965,0.969
-0,102000,0.966,0.969
-0,103000,0.966,0.97
-0,104000,0.965,0.97
-0,105000,0.968,0.969
-0,106000,0.966,0.969
-0,107000,0.966,0.968
-0,108000,0.965,0.969
-0,109000,0.966,0.969
-0,110000,0.964,0.969
-0,111000,0.964,0.969
-0,112000,0.965,0.968
-0,113000,0.965,0.968
-0,114000,0.964,0.969
-0,115000,0.966,0.97
-0,116000,0.967,0.97
-0,117000,0.966,0.97
-0,118000,0.966,0.969
-0,119000,0.964,0.969
-0,120000,0.966,0.969
-0,121000,0.966,0.97
-0,122000,0.967,0.97
-0,123000,0.966,0.97
+0,1000,0.97,0.967
+0,2000,0.97,0.968
+0,3000,0.97,0.967
+0,4000,0.97,0.968
+0,5000,0.971,0.967
+0,6000,0.97,0.966
+0,7000,0.971,0.968
+0,8000,0.97,0.968
+0,9000,0.971,0.966
+0,10000,0.969,0.968
+0,11000,0.97,0.967
+0,12000,0.967,0.967
+0,13000,0.969,0.965
+0,14000,0.971,0.966
+0,15000,0.971,0.967
+0,16000,0.97,0.965
+0,17000,0.969,0.965
+0,18000,0.967,0.967
+0,19000,0.969,0.966
+0,20000,0.968,0.967
+0,21000,0.968,0.966
+0,22000,0.967,0.965
+0,23000,0.97,0.966
+0,24000,0.969,0.967
+0,25000,0.967,0.967
+0,26000,0.968,0.968
+0,27000,0.967,0.965
+0,28000,0.968,0.966
+0,29000,0.967,0.967
+0,30000,0.968,0.968
+0,31000,0.966,0.967
+0,32000,0.967,0.967
+0,33000,0.969,0.967
+0,34000,0.969,0.967
+0,35000,0.968,0.967
+0,36000,0.969,0.967
+0,37000,0.969,0.967
+0,38000,0.968,0.968
+0,39000,0.969,0.966
+0,40000,0.965,0.968
+0,41000,0.968,0.966
+0,42000,0.968,0.969
+0,43000,0.968,0.969
+0,44000,0.969,0.969
+0,45000,0.966,0.967
+0,46000,0.966,0.968
+0,47000,0.968,0.967
+0,48000,0.966,0.968
+0,49000,0.967,0.968
+0,50000,0.966,0.969
+0,51000,0.967,0.967
+0,52000,0.967,0.968
+0,53000,0.968,0.968
+0,54000,0.968,0.966
+0,55000,0.968,0.967
+0,56000,0.968,0.968
+0,57000,0.968,0.967
+0,58000,0.969,0.967
+0,59000,0.969,0.967
+0,60000,0.97,0.966
+0,61000,0.969,0.968
+0,62000,0.968,0.97
+0,63000,0.969,0.969
+0,64000,0.969,0.968
+0,65000,0.969,0.969
+0,66000,0.968,0.965
+0,67000,0.969,0.968
+0,68000,0.968,0.967
+0,69000,0.968,0.968
+0,70000,0.97,0.968
+0,71000,0.97,0.969
+0,72000,0.967,0.968
+0,73000,0.967,0.968
+0,74000,0.968,0.967
+0,75000,0.969,0.965
+0,76000,0.969,0.968
+0,77000,0.968,0.965
+0,78000,0.968,0.966
+0,79000,0.97,0.966
+0,80000,0.969,0.968
+0,81000,0.969,0.967
+0,82000,0.968,0.967
+0,83000,0.969,0.967
+0,84000,0.968,0.967
+0,85000,0.968,0.969
+0,86000,0.97,0.966
+0,87000,0.968,0.969
+0,88000,0.967,0.97
+0,89000,0.967,0.967
+0,90000,0.967,0.968
+0,91000,0.967,0.967
+0,92000,0.968,0.967
+0,93000,0.971,0.966
+0,94000,0.97,0.968
+0,95000,0.967,0.964
+0,96000,0.967,0.966
+0,97000,0.969,0.964
+0,98000,0.969,0.966
+0,99000,0.969,0.967
+0,100000,0.968,0.967
+0,101000,0.967,0.966
+0,102000,0.967,0.967
+0,103000,0.967,0.967
+0,104000,0.967,0.966
+0,105000,0.966,0.967
+0,106000,0.968,0.968
+0,107000,0.968,0.966
+0,108000,0.967,0.968
+0,109000,0.967,0.967
+0,110000,0.968,0.969
+0,111000,0.967,0.968
+0,112000,0.967,0.968
+0,113000,0.967,0.966
+0,114000,0.966,0.966
+0,115000,0.966,0.967
+0,116000,0.966,0.966

pytorch_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:20f6199b13486aae2679b27b5e8830fe4ec79c7823c346a5227850695ed7a1d6
+oid sha256:6bea01d07d85dc7870dfd9907ca78bac21ee8b1ef81c838f25a88b6494c79b32
 size 498834989

tokenizer_config.json

@@ -4,7 +4,7 @@
   "do_lower_case": false,
   "mask_token": "[MASK]",
   "model_max_length": 1000000000000000019884624838656,
-  "name_or_path": "output/original-en-sv-2022-12-20_16-52-40/",
+  "name_or_path": "output/no-normalize-en-sv-2022-12-26_18-39-42/",
   "never_split": null,
   "pad_token": "[PAD]",
   "sep_token": "[SEP]",