

TITLE
Identify issue in slow torch tests

COMMENTS
5

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
Pull request to try and identify the source of the hangs in the torch slow CI. Torch slow CI was taking three hours per run until a few days ago, and has since jumped to 6+ hours, for an unknown reason. The job ends up being killed as it goes over the timeout, so the resulting time might end up being even larger than six hours.

Example of run that took 3 hours (April 20, 2021): https://github.com/huggingface/transformers/actions/runs/765376348
Example of run that took 6+ hours (April 21, 2021: https://github.com/huggingface/transformers/actions/runs/768949009

Here is an example of a run that took 6+ hours, while completing the full common tests: https://github.com/huggingface/transformers/runs/2443524960?check_suite_focus=true

The common tests took 5h56 minutes to complete, and the pipeline tests took more than 4 hours to complete before being apparently killed by CI, so there was clearly something going wrong here.

In order to investigate the root cause of the issue, opening a PR here. Tests will be conducted on a testing machine with the exact same configuration as the other CI machines. Investigating on a single run, on a single GPU machine.

The approach is discussed with @stas00, who is helping out and offered some of the steps below.

## Step 1

The first step is ensuring this is not an error linked to the machine itself, so we first start by running the job on the machine without changing anything to it. We only add a 240-minute timeout so that it can go on to step 2 if it goes over the 4 hour mark (as we know it should take less than 3 hours to complete)

See run for first step here: https://github.com/huggingface/transformers/runs/2554755801

Edit: First run errored out at 6 hours like on other machines. I do not think it is a setup issue.

## Step 2 (if step 1 doesn't resolve the issue)

The second step is twofold: removing `pytest-xdist` as we do not leverage it (we're using a single worker), and adding `pytest-timeout` with a timeout of 300 seconds.

See run for second step here: https://github.com/huggingface/transformers/runs/2554760360

## Step 3 (if step 1 & 2 don't resolve the issue)

Do a manual run - at the 3 hour mark, it should be hanging.

As it is hanging, try and retrieve information about what is information. For example, with the following:

```
pip install py-spy
# dumps traceback for each thread
sudo py-spy dump --pid PID
```

## Step 4 (if no step above resolves the issue)

The diff between the two jobs (3hr and 6hr) doesn't seem to have anything that would make the tests hang - but reverting to the previous repository state could help us identify the culprit. Diff: https://github.com/huggingface/transformers/compare/95037a1..95dab34

Additionally, Stas identified two difference in dependencies between the two runs:

```
-datasets-1.5.0
+datasets-1.6.0
-nltk-3.6.1
+nltk-3.6.2
```

Those should be investigated at the same time.