

TITLE
[Benchmarks] index

COMMENTS
0

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
This issue is to document the important `transformers` benchmarks in one place, so that they are easy to find.

To add a new benchmark entry post it in an Issue (separately or as a comment in an existing issue) and then link from here. If you have edit rights please add a link directly to this post, otherwise please add a note in the comments and I will update this post. 

Please do not post actual benchmarks in the comments of this Issue. This is only an index. 

Thank you!


## Fastest speed combinations
- [RTX-3090](https://github.com/huggingface/transformers/issues/14608#issuecomment-1005229426)
- [A100](https://github.com/huggingface/transformers/issues/15026#issuecomment-1005227577)


## Precision: fp16 vs bf16 vs tf32 vs fp32
- [RTX-3090](https://github.com/huggingface/transformers/issues/14608#issuecomment-1004390803)
- [A100](https://github.com/huggingface/transformers/issues/15026#issuecomment-1004543189)


## Batch size / gradient accumulation steps 
-  gradient accumulation steps: [RTX-3090](https://github.com/huggingface/transformers/issues/14608#issuecomment-1004392537), [A100](https://github.com/huggingface/transformers/issues/15026#issuecomment-1004592231)
- batch size [RTX-3090](https://github.com/huggingface/transformers/issues/14608#issuecomment-1004470417), [A100](https://github.com/huggingface/transformers/issues/15026#issuecomment-1005033957)
 


## Gradient checkpointing
- [RTX-3090](https://github.com/huggingface/transformers/issues/14608#issuecomment-1004422281)
- [A100](https://github.com/huggingface/transformers/issues/15026#issuecomment-1005034578)


## Optimizers:
- Adam torch vs. apex vs HF vs adafactor: [RTX-3090](https://github.com/huggingface/transformers/issues/14608#issuecomment-1005219385), [A100](https://github.com/huggingface/transformers/issues/15026#issuecomment-1005220263)


## Network / Interconnects:
- [DP/DDP/NVLink](https://github.com/huggingface/transformers/issues/9371#issuecomment-768656711)
