

TITLE
Better regex expression for extracting language code in tokenization_marian.py

COMMENTS
4

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
# Better regex expression for extracting language code in tokenization_marian.py

In `tokenization_marian.py`, the regex `>>.+<<` is used to extract the language token from the sentences leading to the following incorrect tokenization.

```
Example Sentence:
>>hin<< We use cout<< function to print a line in C++.

Current Tokenizer gives:
['<unk>', '▁function', '▁to', '▁print', '▁a', '▁line', '▁in', '▁C', '++', '.', '</s>']

Expected Tokenization:
['>>hin<<', '▁We', '▁use', '▁c', 'out', '<', '<', '▁function', '▁to', '▁print', '▁a', '▁line', '▁in', '▁C', '++', '.', '</s>']
```

This pull request changes the regex to `>>.{3}<<|>>.{3}\_.{4}<<` which covers the 194 language tags in the en-mul model.

@sshleifer 