text: 
  
  TITLE
  Key Error: 'pre-processing' during conversion from tatoeba to Marian model
  
  COMMENTS
  3
  
  REACTIONS
  +1: 1
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  ## Environment info
  
  - `transformers` version: `4.6.0.dev0`
  - Platform: `CentOS Linux release 7.7.1908 (Core)`
  - Python version: `3.8.5`
  - PyTorch version: `1.8.1 + cuda 10.2`
  - Tensorflow version: N/A
  - Using GPU in script?: No
  - Using distributed or parallel set-up in script?: No
  
  ### Who can help
  
  Marian: @patrickvonplaten , @patil-suraj 
  
  ## Information
  
  Model I am using (Bert, XLNet ...): Marian
  
  The problem arises when using:
  * [x] the official example scripts: tatoeba to marian model script
  * [ ] my own modified scripts
  
  The tasks I am working on is:
  * [x] an official GLUE/SQUaD task: machine translation
  * [ ] my own task or dataset
  
  ## To reproduce
  
  
  Following the script from [scripts/tatoeba/README.md ](https://github.com/huggingface/transformers/tree/master/scripts/tatoeba)
  1. 
  ```git clone git@github.com:huggingface/transformers.git
  cd transformers
  pip install -e .
  pip install pandas GitPython wget
  ```
  2. 
  ```
  curl https://cdn-datasets.huggingface.co/language_codes/language-codes-3b2.csv  > language-codes-3b2.csv
  curl https://cdn-datasets.huggingface.co/language_codes/iso-639-3.csv > iso-639-3.csv
  ```
  3. `git clone git@github.com:Helsinki-NLP/Tatoeba-Challenge.git`
  4. `python src/transformers/models/marian/convert_marian_tatoeba_to_pytorch.py --models kor-eng eng-kor --save_dir converted/`
  
  Error message:
  ```
  Traceback (most recent call last):
    File "src/transformers/models/marian/convert_marian_tatoeba_to_pytorch.py", line 1267, in <module>
      resolver = TatoebaConverter(save_dir=args.save_dir)
    File "src/transformers/models/marian/convert_marian_tatoeba_to_pytorch.py", line 58, in __init__
      reg = self.make_tatoeba_registry()
    File "src/transformers/models/marian/convert_marian_tatoeba_to_pytorch.py", line 258, in make_tatoeba_registry
      return [(k, v["pre-processing"], v["download"], v["download"][:-4] + ".test.txt") for k, v in results.items()]
    File "src/transformers/models/marian/convert_marian_tatoeba_to_pytorch.py", line 258, in <listcomp>
      return [(k, v["pre-processing"], v["download"], v["download"][:-4] + ".test.txt") for k, v in results.items()]
  KeyError: 'pre-processing'
  ```
  
  ## Expected behavior
  
  Conversion of the model from Tatoeba to Marian for the chosen language pair with no errors.
  