

TITLE
Gradient accumulation and distributed parallelism will reduce the effect?

COMMENTS
3

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
Reference the code: "transformers/examples/legacy/question-answering/run_squad.py"
I found：
1. normal results without using distributed code, using only gradient accumulation.
2. using the distributed code with the parameter gradient_accumulation_steps=1, the effect is normal.
3. using the distributed code with the parameter gradient_accumulation_steps set to other, the experimental effect is abnormal.

What is going on, please? Thanks!
