

TITLE
After 2nd iteration: always same result when training in a loop

COMMENTS
6

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
I train a BERT model on a binary classification task. I do the training 4 times in a row. With same train and validation data and with the exact same hyperparameters. I use the default seed in TrainerArguments and set no other seeds.

The results of the 2nd, 3rd and 4th iteration are 100% the same. The result of the 1st run is unique. This behavior is 100% reproducible.

It is not clear why this is the case. Since I set a seet and work with same data.

PIP Libs I use (no conda libs):
- sentencepiece-0.1.91 
- tokenizers-0.9.3 
- transformers-3.5.1
- torch-1.7.0+cu101

Screenshot:
![grafik](https://user-images.githubusercontent.com/229382/99275325-a57bf080-282b-11eb-843d-fb8c38dc8e3d.png)

Colab to reproduce:
https://colab.research.google.com/drive/1HjQ7p5AlDY9kcWo7uSXzteh7x1ustEdN?usp=sharing

I already did some facy stuff after each iteration:
```python
    del trainer
    del training_args
    del model
    del config
    del train_result
    del tokenizer
    del labeled_dataset_test
    del labeled_dataset_train
    gc.collect()
    torch.cuda.empty_cache()
    torch.cuda.ipc_collect()
    gc.collect()
    torch.cuda.empty_cache()
    torch.cuda.ipc_collect()  
```

But that does not help. Can someone please help me here and claify whats up?

PS: I do not think that a seed can be the reason. If a seed would be the reason the 1st and 2nd run would also be the same and AFAIK a GPU training is not 100% deterministic. So a small difference would still remain.
