

TITLE
Save underlying BertModel only

COMMENTS
2

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
I currently have a custom model on top of a pretrained BERT Model, which is effectively just dropout and a linear layer on top for classification. I want to be able to train this classifier, updating the underlying weights, but then to save the `BERTModel` underneath (w/o linear classification layer) so that I can load from file and use this as my input to the same custom model. Is there a way I can access and save the underlying transformer to then be used again like this?

Code for clarification:
```
class BERTBinaryClassifier(torch.nn.Module):

    def __init__(self, model_name_or_path: str):
        super(BERTBinaryClassifier, self).__init__()
        self.bert = ModelSelection.get_model(
            model_name=model_name_or_path)
        self.drop = torch.nn.Dropout(p=0.3)
        self.out = torch.nn.Linear(self.bert.config.hidden_size, 2).cuda()

    def forward(self, inputs):
        logits, embs, *_ = self.bert(**inputs)
        output = self.drop(logits)
        return self.out(output), embs
```

Right now I'm just outputting the logits and hidden_state for future usage but I'd like to be able to use this same function to effectively load a `BertModel` like I do at the start here with `self.bert` (which just loads `BertModel.from_pretrained` and save my Pytorch classifier as a whole, separately.

Would it be as simple as say accessing `self.bert.bert` and then saving it that way?