

TITLE
[seq2seq testing] multigpu test run via subprocess

COMMENTS
7

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
This PR is trying to fix the hanging/misbehaving/self-replicating pytest for tests using PL with `gpu>1` (`ddp` backend).

OK, I couldn't figure out how to make `dp` or `ddp_spawn` to work, all kinds of obscure errors inside PL (It doesn't look like these are closely maintained as it's recommended not to use either), so `ddp` it is. I tried to get `dp` to work first, since it doesn't require forking a new process and special handling inside pytest.

Here is a working solution for `ddp`. Bottom line - you have to fork a new process and run the distributed script from it - to get it working with `ddp` - otherwise pytest either hangs or runs itself multiple times, breaking other scripts, a big mess.
I borrow the idea from PL itself https://github.com/PyTorchLightning/pytorch-lightning/blob/master/tests/models/test_gpu.py#L111 - what a better place to find the correct way to test something but from the horse's mouth.

So what I had to do:

* [x] split into `test_seq2seq_examples_multi_gpu.py` as requested
* [x] added `subprocess.Popen` - but then replaced it with the modern `asyncio`- apparently using stdout and stderr pipes with `wait` can still cause a deadlock - but let's see if that works for our needs.
* [x] had to mess with args to correctly convert them into cl args - so many of them! perhaps there is already a helper util that does that - I probably re-invented the wheel
* [x] had to provide a new flag `--overwrite_output_dir` to support multi-gpu processes, as one of the children otherwise creates the output dir and the other fails to do so. instead we create the dir in the parent process.
* [x] there are some issues with accessing module attributes in the distributed env (see details here: https://github.com/huggingface/transformers/issues/5887#issuecomment-695919119) - had to tweak the `lightning_base.py` to not use attributes in 2 accessors. (I didn't check - it's possible I need to adjust other scripts if they use `self.total_steps`) - I'm not 100% sure what is different under ddp - but somehow things behave differently and we have no access to module's attributes unless they are part of the model - see `nn.Module.__getattr__` - might have to do with modules getting pickled. if you have insights I'm all ears.
* [x] the test validation had to be adjust to handle 2 gpus

@sshleifer 

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #5887
