File size: 1,343 Bytes
cb302af |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 |
# ClinicalBERT
<!-- Provide a quick summary of what the model is/does. -->
This model card describes the ClinicalBERT model, which was trained on a large multicenter dataset with a large corpus of 1.2B words of diverse diseases we constructed.
## Pretraining Data
The ClinicalBERT model was trained on a large multicenter dataset with a large corpus of 1.2B words of diverse diseases we constructed.
For more details, see here.
## Model Pretraining
### Pretraining Procedures
The training code can be found [here](https://www.github.com/xxx) and the model was trained on four A100 GPU.
Model parameters were initialized with xxx.
### Pretraining Hyperparameters
We used a batch size of xx, a maximum sequence length of xx, and a learning rate of xx for pre-training our models.
The model was trained for xx steps.
The dup factor for duplicating input data with different masks was set to 5.
All other default parameters were used (xxx).
## How to use the model
Load the model via the transformers library:
```python
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("kimpty/ClinicalBERT")
model = AutoModel.from_pretrained("kimpty/ClinicalBERT")
```
## More Information
Refer to the paper xxx.
## Questions?
Post a Github issue on the xxx repo or email xxx with any questions.
|