| # ERNIE-2.0-large | |
| ## Introduction | |
| ERNIE 2.0 is a continual pre-training framework proposed by Baidu in 2019, | |
| which builds and learns incrementally pre-training tasks through constant multi-task learning. | |
| Experimental results demonstrate that ERNIE 2.0 outperforms BERT and XLNet on 16 tasks including English tasks on GLUE benchmarks and several common tasks in Chinese. | |
| More detail: https://arxiv.org/abs/1907.12412 | |
| ## Released Model Info | |
| This released pytorch model is converted from the officially released PaddlePaddle ERNIE model and | |
| a series of experiments have been conducted to check the accuracy of the conversion. | |
| - Official PaddlePaddle ERNIE repo: https://github.com/PaddlePaddle/ERNIE | |
| - Pytorch Conversion repo: https://github.com/nghuyong/ERNIE-Pytorch | |
| ## How to use | |
| ```Python | |
| from transformers import AutoTokenizer, AutoModel | |
| tokenizer = AutoTokenizer.from_pretrained("nghuyong/ernie-2.0-large-en") | |
| model = AutoModel.from_pretrained("nghuyong/ernie-2.0-large-en") | |
| ``` | |
| ## Citation | |
| ```bibtex | |
| @article{sun2019ernie20, | |
| title={ERNIE 2.0: A Continual Pre-training Framework for Language Understanding}, | |
| author={Sun, Yu and Wang, Shuohuan and Li, Yukun and Feng, Shikun and Tian, Hao and Wu, Hua and Wang, Haifeng}, | |
| journal={arXiv preprint arXiv:1907.12412}, | |
| year={2019} | |
| } | |
| ``` | |