

TITLE
Implement PyTorch and/or TensorFlow sequence classification architectures for causal language models

COMMENTS
18

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
# 🚀 Feature request

The architecture `GPT2ForSequenceClassification` was added in #7501 in PyTorch. It would be great to have it in TensorFlow (cf. issues #7622), but it would also be great to have it for other causal models: ~OpenAI GPT~, ~CTRL~ (PR opened @elk-cloner), ~TransfoXL~ (PR opened @spatil6)

Below is a list of items to follow to make sure the integration of such an architecture is complete:

- Implement `XXXForSequenceClassification` in `modeling_xxx.py` or `TFXXXForSequenceClassification` in `modeling_tf_xxx.py
- Test that architecture in `tests/test_modeling_xxx.py` or `tests/test_modeling_tf_xxx.py`
- Add that architecture to `__init__.py` and ` docs/source/model_doc/xxx.rst`.

Taking a look at the code changes in #7501 would be a good start.

A very good first issue to get acquainted with the library and its architectures!
