

TITLE
Conversion of BPE tokenizer for Marian models

COMMENTS
2

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
Hello,

I was searching for the pt-en model of Marian and noticed that it has not been converted for the huggingface library apparently because it uses a BPE tokenizer. Is it possible to convert BPE-based models to be used in huggingface somehow?

Thank you