

TITLE
Add OFA to transformers

COMMENTS
3

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 3
heart: 8
rocket: 0
eyes: 0

BODY
# 🌟 New model addition
We recently proposed OFA, a unified model for multimodal pretraining, which achieves multiple SoTAs on downstream tasks, including image captioning, text-to-image generation, referring expression comprehension, etc. We would like to implement OFA on transformers if it is possible.

## Model description
OFA is a unified multimodal pretrained model that unifies modalities (i.e., cross-modality, vision, language) and tasks (e.g., image generation, visual grounding, image captioning, image classification, text generation, etc.) to a simple sequence-to-sequence learning framework. For more information, please refer to our paper: [Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](http://arxiv.org/abs/2202.03052).

<!-- Important information -->
Our code base is Fairseq. We wonder if there is any simpler solution for the transfer from Fairseq to transformers. To email us, please contact zheluo.wp@alibaba-inc.com or junyang.ljy@alibaba-inc.com.

## Open source status

* [x] the model implementation is available: ([OFA official repo](https://github.com/OFA-Sys/OFA))
* [x] the model weights are available: ([OFA checkpoints](https://github.com/OFA-Sys/OFA/blob/main/checkpoints.md))
* [x] who are the authors: (Peng Wang @[logicwong](https://github.com/logicwong), An Yang @[yangapku](https://github.com/yangapku), Rui Men @[jxst539246](https://github.com/jxst539246), Junyang Lin @[JustinLin610](https://github.com/JustinLin610), Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, Hongxia Yangf)
