

TITLE
Ability to pre-train BART model

COMMENTS
1

REACTIONS
+1: 2
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
# 🚀 Feature request

Ability to pre-train BART model, same like there is an ability to pre-train BERT and other models.

## Motivation

I'm using a pre-trained BART model for sequence-to-sequence problem and trained it using my own data, using the examples here: https://github.com/huggingface/transformers/tree/master/examples/seq2seq

I was wondering if there is a chance to add an ability to continue the pre-training of the already pre-trained `facebook/bart-base` and `facebook/bart-large` models, with my own unsupervised data, in order to improve the results.

@sshleifer Can you please help? thanks in advance!
