Evaluation metric
Can anyone help me calculate the Rouge score while training so that I can get an evaluation after each epoch?
Also, for this model, I'm not getting good performance for QA dataset even after 600 epochs.
My QA format is "Question. Answer: answer".
Hello @codegood !
Since you are fine-tuning a base model, try playing around with the batch size or the learning rate. Usually, when the fine-tuning is aggressive, the model starts forgetting its previous learning and try to adapt to the new data, which would explain a relatively higher loss.
Regarding on adding ROUGE score while training/evaluation, you can check the compute_metrics() from the Trainer class. It allows you to add a custom function and report any desired behavior.
PS: 600 epochs might be an overkill because you would be super overfitting the model to the data. Maybe try also adding new sources of data just to add some sort of regularization.
Best regards,
Gustavo.