init

README.md

@@ -0,0 +1,44 @@
+---
+language: zh
+---
+
+# ERNIE-3.0-base-zh
+
+## Introduction
+
+ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
+More detail: https://arxiv.org/abs/2107.02137
+
+## Released Model Info
+
+This released pytorch model is converted from the officially released PaddlePaddle ERNIE model and 
+a series of experiments have been conducted to check the accuracy of the conversion.
+
+- Official PaddlePaddle ERNIE repo:https://paddlenlp.readthedocs.io/zh/latest/model_zoo/transformers/ERNIE/contents.html
+- Pytorch Conversion repo:  https://github.com/nghuyong/ERNIE-Pytorch
+
+## How to use
+If you want to use ernie-3.0 series models, you need to add `task_type_id` to BERT model following this [MR](https://github.com/huggingface/transformers/pull/18686/files) 
+**OR** you can re-install the transformers from my changed branch.
+```bash
+pip uninstall transformers # optional
+pip install git+https://github.com/nghuyong/transformers@add_task_type_id # reinstall
+```
+Then you can load ERNIE-3.0 model as before:
+```Python
+from transformers import BertTokenizer, BertModel
+
+tokenizer = BertTokenizer.from_pretrained("nghuyong/ernie-3.0-base-zh")
+model = BertModel.from_pretrained("nghuyong/ernie-3.0-base-zh")
+```
+
+## Citation
+
+```bibtex
+@article{sun2021ernie,
+  title={Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation},
+  author={Sun, Yu and Wang, Shuohuan and Feng, Shikun and Ding, Siyu and Pang, Chao and Shang, Junyuan and Liu, Jiaxiang and Chen, Xuyi and Zhao, Yanbin and Lu, Yuxiang and others},
+  journal={arXiv preprint arXiv:2107.02137},
+  year={2021}
+}
+```

config.json

@@ -0,0 +1,21 @@
+{
+    "attention_probs_dropout_prob": 0.1,
+    "hidden_act": "gelu",
+    "hidden_dropout_prob": 0.1,
+    "hidden_size": 768,
+    "initializer_range": 0.02,
+    "max_position_embeddings": 2048,
+    "num_attention_heads": 12,
+    "num_hidden_layers": 12,
+    "task_type_vocab_size": 3,
+    "type_vocab_size": 4,
+    "use_task_id": true,
+    "vocab_size": 40000,
+    "init_class": "ErnieModel",
+    "layer_norm_eps": 1e-05,
+    "model_type": "bert",
+    "architectures": [
+        "BertForMaskedLM"
+    ],
+    "intermediate_size": 3072
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2a060b11ea32dca2c35f9f0fc27539a80109ea323928569841fae6824c94587e
+size 474372903