text: 
  
  TITLE
  rfc: integration tests need non-example application for testing
  
  COMMENTS
  3
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  # 🚀 Feature request
  
  We have an ongoing conflict with some of the core integration tests needing a serious program to be tested with. The only place these can be found is under `examples/` - and so the tests - e.g. deepspeed/apex/fairscale reside under `examples/` because of that.
  The problem is that because they are under `examples/` they are being treated as such, but they are not examples.
  
  I propose we have at least one complex representative example turned into a serious program that is being supported like any other core function. Such program or programs can then be used for integration testing and what we keep on discussing but not getting to do is performance regression. You can't do performance regression on mock ups. 
  
  The few attempts to measure bleu scores are being killed as well with the current `seq2seq` wipeout. How can one do regression testing if there is nothing to measure. The regressions can be subtle and not detected by general common tests. It's a way easier to know that this input should give this bleu score on this model and if it doesn't then something is wrong.
  
  @patrickvonplaten, @sgugger, @LysandreJik, @patil-suraj  