

TITLE
[s2s] Marian beam search slow for en-de 

COMMENTS
8

REACTIONS
+1: 1
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
Hey @sshleifer (tagging because of translation) - I'm not sure whether I am misunderstanding something or this is an actual issue so apologies, but it seems like validation/eval is significantly slower than training and is a serious bottleneck when fine-tuning translation models.

I am running on Colab with V100, and trying to finetune a MarianMT model on a data set of ~10k sentences of length of up to 300, training on 90% of the data takes about 2 minutes per epoch whereas validation/eval on the remaining 10% of the data takes about 6-7 minutes without any fancy eval flags, no beamsearch etc. This results in fine-tuning being ~5x slower if performing validation every epoch (and still significantly slower even if only performing partial validation or every other epoch etc).

I am using apex and pytorch 1.5.1 as instructed in the readme and in the issues regarding apex fp16 training and bs=16 for both train and validation, different batch sizes did not seem to help. Happy to post more info but the rest is pretty similar to the seq2seq examples.