text: 
  
  TITLE
  [Deepspeed zero3] lazy weights init 
  
  COMMENTS
  0
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  I'm pretty sure we need to follow up to the lazy weights init feature https://github.com/huggingface/transformers/pull/11471
  and add under zero3 `deepspeed.zero.GatheredParameters` here (or inside `_init_weights`):
  
  https://github.com/huggingface/transformers/pull/11471/files#diff-6b72b98c4c2dcfc6cc606843917733f5d858374fbc22a735ff483bbc0c1e63eaR1275-R1276
  
  plus need a test.