

TITLE
[RFC] Amphere/tf32 defaults for transformers

COMMENTS
0

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 1
rocket: 0
eyes: 0

BODY
## Background

It's possible to use the new TF32 format automatically when doing fp32 processing for a ~3x speed up, without doing any changes to the code, other than flipping the switch on. But the speed up may come at a cost of accuracy. You can see the differences between the formats in the following image:

![tf32-Mantissa-chart-hi-res-FINAL-400x255](https://user-images.githubusercontent.com/10676103/142496315-d762a485-22d9-459e-8e22-8acf845f64db.png)

You can see that both TF32 and FP32 have the same dynamic range (the magnitude of numbers), but the former has a much lower precision, which depending on a situation may or may not impact the final outcome.

## Emerging Need

As Amphere hardware is emerging and TF32 automatic enabling seems to be going in the direction of being disabled by default (probably starting from pt-1.11?) as discussed here: https://github.com/pytorch/pytorch/issues/67384, we need to communicate to our users how to turn it on/off and what are the impacts on speed and accuracy might be.

Having it on could bring a ~3x speed improvement, and most likely according to the NVIDIA engineers the training quality shouldn't be impacted. But we don't have our first hand experiences yet to provide pragmatic recommendations.

## Available Guides

The on/off machinery is explained here: https://pytorch.org/docs/master/notes/cuda.html#tf32-on-ampere

The other crucial document is: https://pytorch.org/docs/master/notes/numerical_accuracy.html

TF32 Educational blog post: https://blogs.nvidia.com/blog/2020/05/14/tensorfloat-32-precision-format/

## Plan of action

So this issue is both an RFC and is also documenting the need to update: https://huggingface.co/transformers/performance.html

I trust that the commentary will emerge once you start experimenting with the new hardware.

@sgugger, @LysandreJik, @patil-suraj, @patrickvonplaten 
