

TITLE
request about deepspeed tutorial 

COMMENTS
2

REACTIONS
+1: 1
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
Dear @stas00 

You have created this great tutorial here that without you it was really very hard near impossible to be able to train these large models, thank you so much 

https://github.com/huggingface/transformers/issues/8771

Do you mind updating your comment, including how numbers would change if you add distributed training and deepspeed and how the command should be in this case? so to further speedup deepspeed on multiple GPUs 

I also get this message with deepspeed with transformer 4.3.3 running the tutorial:
[deepscale] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 131072.0, reducing to 65536.0

could you tell me if this shows any issue in training? sorry I am very unfamiliar with deepspeed.

One more question, in tutorial you did not use --fp16, could you add a few comments if we can use it with deepspeed?
this is such a great toturial and would be great if we could have all info in one place.

thank you so much for the hard work and a lot of help

