

TITLE
[deepspeed / m2m_100] make deepspeed zero-3 work with layerdrop

COMMENTS
1

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
Same as I had to fix in `wav2vec2` it looks that this fix should eventually go to all models that use `LayerDrop`. At least at the moment Deepspeed is not capable of randomly skipping layers, so this PR uses the same now well tested workaround I used in `wav2vec2`, where all layers always run when deepspeed zero-3 is detected, but the results are ignored if it was meant to be skipped.

https://github.com/huggingface/transformers/blob/69233cf03be5fbce0492f3997e139c4d05499e27/src/transformers/models/wav2vec2/modeling_wav2vec2.py#L817-L849

Perhaps one day Deepspeed will be able to randomly skip layers, at the moment the solution is not the most efficient one. I made a [request](https://github.com/microsoft/DeepSpeed/issues/1888).

When ZeRO-3 is not used the original code path is taken.

The test exercising this code path will be merged as part of this huge additional tests set PR https://github.com/huggingface/transformers/pull/12695 (it's been long overdue).

For posterity, the error for this issue will look something like:

```
RuntimeError: tracing error at step 42: expected the next 2 parameters in the parameter fetch queue to be 
({'id': 26, 'status': 'AVAILABLE', 'numel': 1024, 'ds_numel': 1024, 'shape': (1024,), 'ds_shape': (1024,), 'requires_grad': True, 'grad_shape': None, 'persist': True, 'active_sub_modules': {24}}, {'id': 27, 'status': 'AVAILABLE', 'numel': 1024, 'ds_numel': 1024, 'shape': (1024,), 'ds_shape': (1024,), 'requires_grad': True, 'grad_shape': None, 'persist': True, 'active_sub_modules': {24}}) 
but got 
({'id': 115, 'status': 'NOT_AVAILABLE', 'numel': 0, 'ds_numel': 1024, 'shape': (0,), 'ds_shape': (1024,), 'requires_grad': True, 'grad_shape': None, 'persist': True, 'active_sub_modules': set()}, {'id': 116, 'status': 'NOT_AVAILABLE', 'numel': 0, 'ds_numel': 1048576, 'shape': (0,), 'ds_shape': (1024, 1024), 'requires_grad': True, 'grad_shape': None, 'persist': False, 'active_sub_modules': set()}).
```

Fixes: https://github.com/huggingface/transformers/issues/16688

@patil-suraj, @sgugger 
