

TITLE
Undocumented distributed inference behaviour for `run_summarization.py`

COMMENTS
4

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
### System Info

```shell
Fails with error


Traceback (most recent call last):
  File "/scratches/neuron/anaconda3/envs/T5DST-SGD/bin/transformers-cli", line 5, in <module>
    from transformers.commands.transformers_cli import main
  File "/scratches/neuron/anaconda3/envs/T5DST-SGD/lib/python3.8/site-packages/transformers/commands/transformers_cli.py", line 26, in <module>
    from .user import UserCommands
  File "/scratches/neuron/anaconda3/envs/T5DST-SGD/lib/python3.8/site-packages/transformers/commands/user.py", line 20, in <module>
    from huggingface_hub.hf_api import HfFolder, create_repo, list_repos_objs, login, logout, whoami
ImportError: cannot import name 'list_repos_objs' from 'huggingface_hub.hf_api' (/scratches/neuron/anaconda3/envs/T5DST-SGD/lib/python3.8/site-packages/huggingface_hub/hf_api.py)


However I am running `4.16.2` with python `3.8`.
```


### Who can help?

@sgugger @stevhliu @patil-suraj 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

I am working with a copy of the `run_summarization.py` (pytorch) example that the authors of this [paper](https://arxiv.org/pdf/2109.07506.pdf) modified to work for dialogue state tracking (implemented [here ](https://github.com/chiahsuan156/DST-as-Prompting) for reference)

The `run_summarization.py` script can be launched with `torch.distributed.launch` and the `--do_predict` option. This shards the examples in the test set to various GPUs and therefore generation and task-oriented metrics is accelerated. The predictions are written to the `generated_predictions.txt` file in the output directory. 

To be able to compute dialogue-relevant task oriented metrics, one ought to run a postprocessing script that uses the `generated_predictions.txt`. Because the trainer erases all the columns that are not keys to the model `forward` method from the dataset, the metadata that informs us of what training examples the predictions are related to is lost. Therefore, we rely on the ordering of the `generated_predictions.txt` to match the order of the examples in the dataset.

My question is: 
 - Does `predictions` (`L675`)  obey the order of the `dataset`? So if my dataset has 1m examples, will the 1m entries in the `predictions` list match the order of the dataset iterator? In my experience this depends on implementation* and the behaviour is not documented.
 
*For example, in frameworks such as `ray` you have to explicitly enforce the order in which the results are returned and the predictions may be returned out of order - if a process finishes, it returns its results so it can be given more work by an external load balancer.

### Expected behavior

```shell
Improved documentation about expected behaviour here. Happy to discuss where this should be added and contribute a small PR to clarify this important issue.
```
