Create README.md

README.md

@@ -0,0 +1,32 @@
+# ChineseBERT-large
+
+This repository contains code, model, dataset for **ChineseBERT** at ACL2021.
+
+paper:  
+**[ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information](https://arxiv.org/abs/2106.16038)**  
+*Zijun Sun, Xiaoya Li, Xiaofei Sun, Yuxian Meng, Xiang Ao, Qing He, Fei Wu and Jiwei Li*
+
+code:   
+[ChineseBERT github link](https://github.com/ShannonAI/ChineseBert)
+
+## Model description
+We propose ChineseBERT, which incorporates both the glyph and pinyin information of Chinese
+characters into language model pretraining.  
+ 
+First, for each Chinese character, we get three kind of embedding.
+ - **Char Embedding:** the same as origin BERT token embedding.
+ - **Glyph Embedding:** capture visual features based on different fonts of a Chinese character.
+ - **Pinyin Embedding:** capture phonetic feature from the pinyin sequence ot a Chinese Character.
+ 
+Then, char embedding, glyph embedding and pinyin embedding 
+are first concatenated, and mapped to a D-dimensional embedding through a fully 
+connected layer to form the fusion embedding.   
+Finally, the fusion embedding is added with the position embedding, which is fed as input to the BERT model.  
+The following image shows an overview architecture of ChineseBERT model.
+ 
+![MODEL](https://raw.githubusercontent.com/ShannonAI/ChineseBert/main/images/ChineseBERT.png)
+
+ChineseBERT leverages the glyph and pinyin information of Chinese 
+characters to enhance the model's ability of capturing
+context semantics from surface character forms and
+disambiguating polyphonic characters in Chinese.