Is Spam all we need? A RoBERTa Based Approach To Spam Detection

Intro

This is inspired largely by mshenoda's roberta spam huggingFace model (https://huggingface.co/mshenoda/roberta-spam).

However, instead of fine-tuning it on all the data sources that the original author had, I only finetuned using the telegram and enron spam/ham datasets. The idea behind this was a more diversified data source, preventing overfitting to the original distribution, and just a fun NLP exploratory experiment. This was fine-tuned by replicating the sentiment analysis Google collab example provided in the Roberta resources page (https://huggingface.co/docs/transformers/main/en/model_doc/roberta#resources) Google collab example.

NOTE: This was done for an interview project, so if you find this by chance... hopefully it helps you too, but know there's definitely better resources out there... and that this was done in the span of one evening.

Metrics

Accuracy: 0.9503 Thrilling, I know, I also just got the chills, especially since my performance is arguably worse than the original authors 😂

Granted, I only ran it for one epoch, and the data is taken from different distributions. I'm sure it would've been more "accurate" if I had just trained it on the SMS data, but diversity is good. And, it's fun to see how stuff impacts the final result!

Model Output

0 is ham
1 is spam

Dataset(s)

The dataset is composed of messages labeled by ham or spam (0 or 1), merged from two data sources:

Telegram Spam Ham https://huggingface.co/datasets/thehamkercat/telegram-spam-ham/tree/main
Enron Spam: https://huggingface.co/datasets/SetFit/enron_spam/tree/main (only used message column and labels)

The dataset used for testing was the original kaggle competition (as part of the interview project that this was for)

SMS Spam Collection https://www.kaggle.com/datasets/uciml/sms-spam-collection-dataset

Dataset Class Distribution

	Total	Training	Testing
Counts	59267	53693 (90.6% )	5574 (9.4%)

	Total	Spam	Ham	Set	% Total
Enron	33345	16852 (50.5%)	16493 (49.5%)	Train	56.2%
Telegram	20348	6011 (29.5%)	14337 (70.5%)	Train	43.8%
SMS	5574	747 (13.5%)	4827 (86.5%)	Test	100%

	Distribution of number of characters per class label (100 bins)	Distribution of number of words per class label (100 bins)
SMS
Enron (limiting a few outliers)
Telegram

^ Note the tails, very interesting distributions. But more so, good to see Benford's law is alive and well in these.

Architecture

The model is fine tuned RoBERTa

roberta-base: https://huggingface.co/roberta-base

paper: https://arxiv.org/abs/1907.11692

Code

https://huggingface.co/ggrizzly/roBERTa-spam-detection/resolve/main/roberta_spam_classifier_fine_tuning_google_collab.ipynb

ggrizzly
/

roBERTa-spam-detection