

TITLE
Model Parallelism and Big Models

COMMENTS
64

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 1
heart: 13
rocket: 6
eyes: 0

BODY
# 🚀 Feature request

This is a discussion issue for training/fine-tuning very large transformer models. Recently, model parallelism was added for gpt2 and t5. The current implementation is for PyTorch only and requires manually modifying the model classes for each model. Possible routes (thanks to @stas00  for identifying these):
- `fairscale` to avoid individual model implementation
- `deepspeed` to possibly enable even larger models to be trained