

TITLE
[Benchmark] V100/A100 benchmarks, dashboard concept

COMMENTS
7

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 2
rocket: 0
eyes: 0

BODY
# 🖥 Benchmarking `transformers`

I benchmarked a couple of models (training) on V100 and A100 and wrapped everything into a Streamlit dashboard [link here](https://share.streamlit.io/tlkh/transformers-benchmarking/main/app.py).

This dashboard shows the measured performance of GPUs when training various configurations of Transformer networks, showing throughput (seq/s) and GPU memory (VRAM) usage. The idea is to allow users have an easy reference for choosing model configuration (model size/batch size/sequence length) and GPU model.

This is kind of a weekend project done out of curiosity. If there is potential, perhaps a more serious effort can be undertaken here. 

## Benchmark

Which part of `transformers` did you benchmark?

Model training: `distilroberta-base`, `roberta-base`, `roberta-large` via `AutoModelForSequenceClassification`

## Set-up

What did you run your benchmarks on? Please include details, such as: CPU, GPU? If using multiple GPUs, which parallelization did you use?

GPUs: V100 16GB, A100 40GB

Single GPU only.

More information and code: https://github.com/tlkh/transformers-benchmarking

## Results

https://share.streamlit.io/tlkh/transformers-benchmarking/main/app.py
