

TITLE
RFC: `torch==1.12` will toggle `torch.backends.matmul.allow_tf32` to `False`  - what should we do?

COMMENTS
14

REACTIONS
+1: 1
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
Ampere GPUs added a new mode called [TF32](https://developer.nvidia.com/blog/accelerating-ai-training-with-tf32-tensor-cores/). Pytorch created a new flag to support the TF32 mode enabling using `torch.backends.matmul.allow_tf32` which has been `True` by default in pytorch since it was added. 

Having this mode on means that matrix multiplications when inputs were in FP32 were actually done in TF32, which made the math significantly faster, albeit less precise (TF32 has the dynamic range of BF16, and the precision of FP16).

The NVIDIA engineers have done many experiments and have found that Deep Learning training accuracy doesn't get impacted for worse by using TF32 instead of FP32 (and often is better), but it provides a significant speed up. It's easy to see from the [A100 spec](https://www.nvidia.com/en-us/data-center/a100/) why:
```
FP32 |  19.5 TFLOPS
TF32 | 156   TFLOPS 
```
(numbers with no sparsity)

And the accuracy tables are:
![AI_training_TF32_tensor_cores_F3-1024x565](https://user-images.githubusercontent.com/10676103/161663313-b6913317-36f2-45c6-94a4-d70dcce2b301.png) from [Accelerating AI Training with NVIDIA TF32 Tensor Cores](https://developer.nvidia.com/blog/accelerating-ai-training-with-tf32-tensor-cores/)

However, the lost precision for some non-DL applications is a problem. Therefore starting from pytorch 1.12 (already in nightly shortly) the default for `torch.backends.matmul.allow_tf32` will be `False`, which won't make the training accuracy worse, but it'll make fp32 training significantly slower. So if you believe we should remain consistent/back compatible - most likely we should turn it back on for pt>1.11:

```
if version.parse(torch.__version__) > version.parse("1.11"):
    torch.backends.matmul.allow_tf32 = True
```
at a single point which always gets executed for pytorch users.

The question is whether this should be done:
1. Not at all - let the user sort it out
2. Transformers-wide
3. Only in HF Trainer (and Accelerate) and if not done add a new flag to let the user control the behavior 

Additionally other use-modes should be made in sync:
1. PyTorch/XLA (some other flag?)

Currently tf32 and how to flip it on/off is documented here: https://huggingface.co/docs/transformers/performance#tf32

A detailed discussion with multiple links to other related resources is here: https://dev-discuss.pytorch.org/t/pytorch-and-tensorfloat32/504

@LysandreJik, @sgugger, @patrickvonplaten, @patil-suraj 