

TITLE
Are there memory leaks when using DeepSpeed on training T5?

COMMENTS
5

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
We’ve been pretraining a T5-small model from scratch with using DeepSpeed v0.3.10.
We’ve found that cpu memory was increasing over time. (We’ve trained about 150 hours)
Are there memory leaks when using DeepSpeed on training T5?
