text: 
  
  TITLE
  [seq2seq] some logging for all processes in distributed mode
  
  COMMENTS
  15
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  In 2D Parallelism, e.g. Pipeline + DeepSpeed I need to log unique device maps per process for the user to see, but currently `logger.info()` is only activated for the main process via `if is_main_process`. Currently only in `examples/seq2seq/run_seq2seq.py`, `examples/seq2seq/finetune_trainer.py`, but it'll be needed for other scripts as well down the road.
  
  Any idea how I could accomplish that while keeping things as they are? I guess I could use `logger.warn` as a workaround, since it's not disabled for other processes. But it's not a good approach, since it's a WARNING after all. And I don't quite want to use `print()` as it might not be what the user wants if they want things quiet.
  
  Perhaps you have some other ideas on how I could go about doing that. 
  
  I think perhaps adding another logger that's INFO-activated for all distributed processes, and is used only occasionally when the normal logger won't do.
  
  I think as we are getting more and more into distributed training we will need to be able to log specific things for specific processes.
  
  Thank you.
  
  @LysandreJik, @patrickvonplaten, @sgugger 