

TITLE
CLIP

COMMENTS
5

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
# What does this PR do?

This PR adds the [CLIP](https://github.com/openai/CLIP) model.

CLIP is a multi-modal vision+language model which uses a transformer model for encoding both the images and text.

- The model here is designed such that both `CLIPTextModel` and `CLIPVisionModel` can be loaded independently, and composed together to get the `CLIPModel`.
- Both `CLIPTextModel` and `CLIPVisionModel` use the shared encoder class `CLIPEncoder`.
- The config classes are also kept in separate i.e `CLIPTextConfig` and `CLIPVisionConfig`.  This could be kept in one config class but then we would have to add two arguments for each config value i.e `text_hidden_size` for text model `vision_hidden_size` for vision model etc.
One issue here is that when we load an individual model, like `CLIPTextModel` using the weights of the whole `CLIPModel`
the config ends up containing both text and vision config dicts, this does not cause any issue but could be confusing to look at. 

One important thing to note here is that CLIP's tokenizer does have a pad token defined for it, but they use 0 as `pad_token_id`  to pad the text, but the token, but the token associated with 0 is not a pad token. So here, to able to do padding I've added `pad_token_id` as a `property` which returns 0. I would be happy to hear if there is some other way to achieve this.

Also, I've added a processor class here but not sure if we really need it for this model. We could easily use the extractor for the vision model and tokenizer for the text model.

Would love your review about the design @LysandreJik , @patrickvonplaten , @sgugger.