Update README.md

README.md

@@ -1,3 +1,104 @@
 ---
 license: cc-by-nc-sa-4.0
 ---
+
+# DRAGON RoBERTa base domain-specific
+
+Pretrained model on Dutch clinical reports using a masked language modeling (MLM) objective. It was introduced in [this](TODO: add link) paper.The model was pretrained using domain-specific data (i.e., clinical reports) from scratch. The architecture is the same as [`xlm-roberta-base`](https://huggingface.co/xlm-roberta-base) from HuggingFace. The tokenizer was fitted to the dataset of Dutch medical reports, using the same settings for the tokenizer as [`roberta-base`](https://huggingface.co/FacebookAI/roberta-base).
+
+
+
+## Model description
+RoBERTa is a transformers model that was pretrained on a large corpus of Dutch clinical reports in a self-supervised fashion. This means it was pretrained on the raw texts only, with no humans labeling them in any way with an automatic process to generate inputs and labels from those texts. More precisely, it was pretrained with the MLM objective:
+
+Masked language modeling (MLM): taking a sentence, the model randomly masks 15% of the words in the input then runs the entire masked sentence through the model and has to predict the masked words. This is different from traditional recurrent neural networks (RNNs) that usually see the words one after the other, or from autoregressive models like GPT which internally masks the future tokens. It allows the model to learn a bidirectional representation of the sentence.
+
+This way, the model learns an inner representation of the Dutch medical language that can then be used to extract features useful for downstream tasks: if you have a dataset of labeled reports, for instance, you can train a standard classifier using the features produced by the BERT model as inputs.
+
+## Model variations
+Multiple architectures were pretrained for the DRAGON challenge. 
+
+| Model | #params | Language |
+|------------------------|--------------------------------|-------|
+| [`joeranbosma/dragon-bert-base-mixed-domain`](https://huggingface.co/joeranbosma/dragon-bert-base-mixed-domain) | 109M   | Dutch → Dutch |
+| [`joeranbosma/dragon-roberta-base-mixed-domain`](https://huggingface.co/joeranbosma/dragon-roberta-base-mixed-domain) | 278M    | Multiple → Dutch |
+| [`joeranbosma/dragon-roberta-large-mixed-domain`](https://huggingface.co/joeranbosma/dragon-roberta-large-mixed-domain) | 560M    | Multiple → Dutch |
+| [`joeranbosma/dragon-longformer-base-mixed-domain`](https://huggingface.co/joeranbosma/dragon-longformer-base-mixed-domain) | 149M    |  English → Dutch |
+| [`joeranbosma/dragon-longformer-large-mixed-domain`](https://huggingface.co/joeranbosma/dragon-longformer-large-mixed-domain) | 435M    | English → Dutch |
+| [`joeranbosma/dragon-bert-base-domain-specific`](https://huggingface.co/joeranbosma/dragon-bert-base-domain-specific) | 109M | Dutch |
+| [`joeranbosma/dragon-roberta-base-domain-specific`](https://huggingface.co/joeranbosma/dragon-roberta-base-domain-specific) | 278M | Dutch |
+| [`joeranbosma/dragon-roberta-large-domain-specific`](https://huggingface.co/joeranbosma/dragon-roberta-large-domain-specific) | 560M | Dutch |
+| [`joeranbosma/dragon-longformer-base-domain-specific`](https://huggingface.co/joeranbosma/dragon-longformer-base-domain-specific) | 149M | Dutch |
+| [`joeranbosma/dragon-longformer-large-domain-specific`](https://huggingface.co/joeranbosma/dragon-longformer-large-domain-specific) | 435M | Dutch |
+
+
+## Intended uses & limitations
+You can use the raw model for masked language modeling, but it's mostly intended to be fine-tuned on a downstream task.
+
+Note that this model is primarily aimed at being fine-tuned on tasks that use the whole text (e.g., a clinical report) to make decisions, such as sequence classification, token classification or question answering. For tasks such as text generation you should look at model like GPT2.
+
+## How to use
+You can use this model directly with a pipeline for masked language modeling:
+
+```python
+from transformers import pipeline
+unmasker = pipeline("fill-mask", model="joeranbosma/dragon-roberta-base-domain-specific")
+unmasker("Dit onderzoek geen aanwijzingen voor significant carcinoom. PIRADS <mask>.")
+```
+
+Here is how to use this model to get the features of a given text in PyTorch:
+
+```python
+from transformers import AutoTokenizer, AutoModel
+tokenizer = AutoTokenizer.from_pretrained("joeranbosma/dragon-roberta-base-domain-specific")
+model = AutoModel.from_pretrained("joeranbosma/dragon-roberta-base-domain-specific")
+text = "Replace me by any text you'd like."
+encoded_input = tokenizer(text, return_tensors="pt")
+output = model(**encoded_input)
+```
+
+## Limitations and bias
+Even if the training data used for this model could be characterized as fairly neutral, this model can have biased predictions. This bias will also affect all fine-tuned versions of this model.
+
+## Training data
+For pretraining, 4,333,201 clinical reports (466,351 consecutive patients) were selected from Ziekenhuisgroep Twente from patients with a diagnostic or interventional visit between 13 July 2000 and 25 April 2023. 180,439 duplicate clinical reports (179,808 patients) were excluded, resulting in 4,152,762 included reports (463,692 patients). These reports were split into training (80%, 3,322,209 reports), validation (10%, 415,276 reports), and testing (10%, 415,277 reports). The testing reports were set aside for future analysis and are not used for pretraining.
+
+## Training procedure
+
+### Pretraining
+The details of the masking procedure for each sentence are the following:
+- 15% of the tokens are masked.
+- In 80% of the cases, the masked tokens are replaced by `[MASK]`.
+- In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
+- In the 10% remaining cases, the masked tokens are left as is.
+
+### Pretraining hyperparameters
+
+The following hyperparameters were used during pretraining:
+- `learning_rate`: 0.0006
+- `train_batch_size`: 16
+- `eval_batch_size`: 16
+- `seed`: 42
+- `gradient_accumulation_steps`: 16
+- `total_train_batch_size`: 256
+- `optimizer`: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- `lr_scheduler_type`: linear
+- `num_epochs`: 10.0
+- `max_seq_length`: 512
+
+### Framework versions
+
+- Transformers 4.29.0.dev0
+- Pytorch 2.0.0+cu117
+- Datasets 2.11.0
+- Tokenizers 0.13.3
+
+## Evaluation results
+
+Pending evaluation on the DRAGON benchmark.
+
+### BibTeX entry and citation info
+
+```bibtex
+@article{PENDING}
+```