PubMedBert-PubMed200kRCT

This model is a fine-tuned version of microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext on the PubMed200kRCT dataset. It achieves the following results on the evaluation set:

• Loss: 0.2833
• Accuracy: 0.8942

Model description

More information needed

Intended uses & limitations

The model can be used for text classification tasks of Randomized Controlled Trials that does not have any structure. The text can be classified as one of the following:

• BACKGROUND
• CONCLUSIONS
• METHODS
• OBJECTIVE
• RESULTS

The model can be directly used like this:

from transformers import TextClassificationPipeline
from transformers import AutoTokenizer, AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained("pritamdeka/PubMedBert-PubMed200kRCT")
tokenizer = AutoTokenizer.from_pretrained("pritamdeka/PubMedBert-PubMed200kRCT")
pipe = TextClassificationPipeline(model=model, tokenizer=tokenizer, return_all_scores=True)
pipe("Treatment of 12 healthy female subjects with CDCA for 2 days resulted in increased BAT activity.")

Results will be shown as follows:

[[{'label': 'BACKGROUND', 'score': 0.0028450002428144217},
  {'label': 'CONCLUSIONS', 'score': 0.2581048607826233},
  {'label': 'METHODS', 'score': 0.015086210332810879},
  {'label': 'OBJECTIVE', 'score': 0.0016815993003547192},
  {'label': 'RESULTS', 'score': 0.7222822904586792}]]

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 64
• eval_batch_size: 64
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 2.0

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
0.3604	0.14	5000	0.3162	0.8821
0.3326	0.29	10000	0.3112	0.8843
0.3293	0.43	15000	0.3044	0.8870
0.3246	0.58	20000	0.3040	0.8871
0.32	0.72	25000	0.2969	0.8888
0.3143	0.87	30000	0.2929	0.8903
0.3095	1.01	35000	0.2917	0.8899
0.2844	1.16	40000	0.2957	0.8886
0.2778	1.3	45000	0.2943	0.8906
0.2779	1.45	50000	0.2890	0.8935
0.2752	1.59	55000	0.2881	0.8919
0.2736	1.74	60000	0.2835	0.8944
0.2725	1.88	65000	0.2833	0.8942

Framework versions

• Transformers 4.18.0.dev0
• Pytorch 1.10.0+cu111
• Datasets 1.18.3
• Tokenizers 0.11.6

Citing & Authors