

TITLE
[Performance] Tracking open Issues and PRs (pytorch transformers)

COMMENTS
0

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 5
rocket: 0
eyes: 0

BODY
Let's use this Issue to track performance issues and enhancement requests, so it's easier to prioritize the work.

**This is for pytorch `transformers`**

Also I will label it as a `Good Difficult Issue` in case someone is ready for a challenging but rewarding experience of figuring things out. If you do want to take the challenge comment in the corresponding Issue/PR that resonates with you so others would know you're working on it.

If I missed any other relevant open performance-related Issues/PRs that need attention please comment below.

## Regression:
- [ ] https://github.com/huggingface/transformers/pull/11218 Regression after Bart-like refactoring - need to compare the original Bart refactoring PR since most likely the regression happened there.
- [ ]

## Odd slowness:
- [ ] https://github.com/huggingface/transformers/issues/10816 figuring out why eval with --fp16_full_eval is 25% slower
- [ ]

## Fused kernels possibilities:
- [ ] https://github.com/huggingface/transformers/issues/11368 Megatron fused CUDA kernels to improve Hugging Face model classes' scalability 
- [ ] research pytorch kernels?
- [ ] I know Deepspeed has various kernels that we might be able to use

## Faster / leaner startup / module loading
- [ ] https://github.com/huggingface/transformers/issues/12274 - skip storage allocation which gets dropped for pretrained weights

## Faster optimizers
- [ ] https://github.com/huggingface/transformers/issues/12084 - a proposal to port `MemoryEfficientFP16Optimizer` from fairseq
- [ ] https://github.com/huggingface/transformers/issues/9965 - `torch.optim._multi_tensor` faster optimizers - having some bottleneck in the test script - need to profile

## Scalability
- [ ] https://github.com/huggingface/transformers/issues/10321 Tensor Parallelism

## Deepspeed-specific features
- [ ] https://github.com/huggingface/transformers/issues/9606 a list of features that can be integrated
- [ ] https://github.com/huggingface/transformers/issues/12273 - make `from_pretrained` loading faster

## Tests
- [ ] No issue yet, but we really need to add performance regression tests

