| # CirBERTa | |
| ### Apply the Circular to the Pretraining Model | |
| | 预训练模型 | 学习率 | batchsize | 设备 | 语料库 | 时间 | 优化器 | | |
| | --------------------- | ------ | --------- | ------ | ------ | ---- | ------ | | |
| | CirBERTa-Chinese-Base | 1e-5 | 256 | 10张3090+3张A100 | 200G | 2月 | AdamW | | |
| 使用通用语料（WuDao 200G) 进行无监督预训练 | |
| 在多项中文理解任务上，CirBERTa-Base模型超过MacBERT-Chinese-Large/RoBERTa-Chinese-Large | |
| ### 加载与使用 | |
| 依托于huggingface-transformers | |
| ``` | |
| from transformers import AutoTokenizer,AutoModel | |
| tokenizer = AutoTokenizer.from_pretrained("WENGSYX/CirBERTa-Chinese-Base") | |
| model = AutoModel.from_pretrained("WENGSYX/CirBERTa-Chinese-Base") | |
| ``` | |
| ### 引用: | |
| (暂时先引用这个，论文正在撰写...) | |
| ``` | |
| @misc{CirBERTa, | |
| title={CirBERTa: Apply the Circular to the Pretraining Model}, | |
| author={Yixuan Weng}, | |
| howpublished={\url{https://github.com/WENGSYX/CirBERTa}}, | |
| year={2022} | |
| } | |
| ``` | |