

TITLE
Unified transformer interface

COMMENTS
3

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
# 🚀 Feature request

As we're called `transformers`,  it would be nice if there's a `Transformer` class that is not associated with any pretrained model and that we can directly instantiate, like
```python
from transformers import Transformer, Seq2SeqTransformer
transformer = Transformer(n_layers=8, dim=512)
seq2seq_transformer = Seq2SeqTransformer(n_enc_layers=6, n_dec_layers=8, dim=512)
```
They could follow a modular design that makes swapping components very easy, e.g.
```python
class SelfAttention:
    ...

class Transformer:
    def __init__(self, self_attention_cls=SelfAttention, n_layers=8, ...):
        ...
```

## Motivation

1. Makes it easier for researchers to modify the transformer architecture and train it from scratch.
2. Allows great refactoring of the current `modeling_xxx.py` files. Often these files are highly identical minus a few minor differences. For example, the embeddings, self-attention layers, etc. are often identical across multiple models, but the current design does not allow much code re-use. With this change, I imagine a lot of the current `modeling_xxx.py` files can be reduced significantly.
3. A side-effect of this refactoring would be increased readability. Without much repetitive code, it would be much clearer to see the differences between architectures, which would be helpful for many users.