

TITLE
[Tensor Parallelism] Megatron-LM to transformers

COMMENTS
7

REACTIONS
+1: 3
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 1

BODY
# 🚀 Feature request

Splitting the discussion that started here: https://github.com/huggingface/transformers/pull/10301#issuecomment-782917393 to add the potential future feature of transformers and it's Tensor Parallelism (Horizontal Model Parallelism) - for bigger context please see [Parallelism notes](https://github.com/huggingface/transformers/issues/9766).

Let's start with important clarification: MP can mean many different things
1. Vertical MP - slice the layers vertically - one or more full layers placed on each gpu = Vertical MP - in which case VertMP is a simple version of PP with chunks=1
2. Horizontal MP - slice the layers horizontally - place a slice of a full model on each gpu - Example Megatron-LM

At the moment I think it's only Megatron-LM that implements Horizontal MP. @anthon-l has ported that model to `transformers`, except the Horizontal MP parts, since currently `transformers` doesn't yet have support for it. There is already naive Vertical MP in t5 and gpt2 thanks to @alexorona's work, I ported Bart too but it's unmerged, and there is an ongoing effort to figure out how to implement the Pipeline. All these will have to co-operate with each other and also share common tools.

@anton-l [started sharing](https://github.com/huggingface/transformers/pull/10301#issuecomment-782917393) what needs to be done to make that important feature available - and then down the road potentially make it available to other (all?) `transformers` models.

@anton-l, the floor is yours.