

TITLE
Better Fine-Tuning by Reducing Representational Collapse

COMMENTS
1

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
# 🚀 Feature request

Add r3f/r4f to some popular objective functions as suggested by [Armen et. al](https://arxiv.org/abs/2008.03156).

## Motivation

Finetuning is a primary use case of many users of the transformers library.
We can use r3f/r4f to reduce representation collapse by vocabulary inefficiencies.
This is also a relatively cheap feature to implement.

## Your contribution

Understand that this is a relatively new paper with not too much benchmarking done. Will do PR if requested.