

TITLE
Load pretrained model except the head layer for a specific downstream task

COMMENTS
6

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
# 🚀 Feature request

It would be nice to have a flag for `from_pretrained` method that indicates whether to load last layer or not. This feature is needed for transfer learning.

## Motivation

I have trained a model with a specific dataset for a downstream task. Now, I need to train another model that needs to be trained on a similar dataset with different labels. I know that previous model have learned the features from the previous dataset and the new model doesn't need to start from scratch. When I try to load the first model with `from_pretrained` method, it returns size mismatch error due to last layer that has different shape for different number of labels. If there is a flag to load/not to load the last layer, I can initialize last layer randomly and go on my training with transfer learning.

