racro/sentiment-analysis-browser-extension

Fine-tuned BERT model

We open source this fine-tuned BERT model to identify critical aspects within user reviews of adblocking extensions. For every user review, the model provides a criticality score (in the range of -1 to 1) with the negative scores signifying higher probability of finding critical topics within in the reviews.

We have used the distilbert-base-uncased as the base model and fine-tuned it on a manually annotated dataset of webstore reviews.

Further details can be found in our AsiaCCS paper - From User Insights to Actionable Metrics: A User-Focused Evaluation of Privacy-Preserving Browser Extensions.

Note: We haven't tested its accuracy on user reviews from other categories but are open to discussing the possibility of extrapolating it to other product categories. Feel free to raise issues in the repo or contact the author directly.

Intended uses & limitations

The model has been released for free use. It has not been trained on any private user data. Please cite the above paper in our works.

Evaluation data

It achieves the following results on the evaluation set:

• Loss: 0.4768
• Accuracy: 0.8615
• F1: 0.8816

Training procedure

The training dataset comprised of 620 reviews and the test dataset had 150 reviews. The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 6

Framework versions

• Transformers 4.34.0
• Pytorch 2.0.1+cu118
• Datasets 2.14.5
• Tokenizers 0.14.1