

TITLE
[trainer]  speed issues: --fp16 doesn't improve speed, DP runs really slow 

COMMENTS
11

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
Splitting from https://github.com/huggingface/transformers/issues/9156#issuecomment-747636108 where while running benchmarks for fairscale's sharded ddp support I noticed that there was almost no difference between having or not having `--fp16` to the training runtime.

Not sure whether this impacts trainer in general or just seq2seq finetune_trainer.py, but there is almost no speed improvements with adding `--fp16`. 

This is with pytorch-nightly.

Not sure whether this has to do with the recent autocast cache blowup fix https://github.com/pytorch/pytorch/issues/48049 (our corresponding issue https://github.com/huggingface/transformers/issues/8403) - part of the fix was to remove some of the cash that perhaps was essential and thus leads to this issue.

I will test with apex and compare.

Also as can be seen from: https://github.com/huggingface/transformers/issues/9179#issuecomment-747783879
DP is running really slow!

@sgugger 
