

TITLE
Supporting multiple evaluation datasets in `Trainer` and `Seq2seqTrainer`

COMMENTS
6

REACTIONS
+1: 1
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
# 🚀 Feature request

Support for evaluating on multiple validation datasets when using the `Trainer` class.

## Motivation

This is a common use case in research. Imagine that you train your model on some data, and you have validation data coming from two distinct distributions. You want to compute two sets of metrics - one for the validation dataset with the same distribution as the training data and one for the validation dataset with known  distribution.

## Your contribution

- Happy to submit an example with my own code (assuming the research makes sense) so that others see how this can be achieved in practice. 

- Could update relevant posts on the huggingface forum so that other users requiring this feature can see how it can be done.

## Things I have tried

Inspired by this post [here](https://discuss.huggingface.co/t/evaluating-your-model-on-more-than-one-dataset/1544) and @sgugger's solution, I set out to see if implementing the `on_evaluate` callback is possible, but I can't figure out how to get the other validation datasets to it - the callaback can only access the trainer init arguments/state but none of these objects can be passed additional data loaders. 

How can this be approached instead? My current solution is not clean, but may work:

1. Use `setattr` to add an attribute to the trainer after init, call it `additional_eval_datasets`
2. Override the `_maybe_log_save_evaluate` method as follows:
         - Call the `Trainer` superclass method first to do what the trainer would normally do
         - loop through the additional datasets, calling the `Trainer.evaluate` for each dataset with appropriate inputs

This is a bit hacky as I should not be overriding a private method, but override `evaluate` instead. However, the implementation would be more concise in this way. Please feedback of any undesirable side effects that this may lead to - I did read through the `Trainer` source code and did not spot any pitfalls!

Of course, this approach can be adapted in order to support evaluation on multiple datasets natively in the `Trainer`.   

