Observations!
I'm sharing results of some experimentation I've done with your model in part for my research. The relevant paper and repository for the experimental data are as follows:
Paper: https://arxiv.org/abs/2402.08309
Repository for Test Data & Tools: https://github.com/nahmiasd/Prompted-Contextual-Vectors-for-Spear-Phishing-Detection
I've found that this model is extremely weak against LLM generated spear-phishing emails, of the 253 I tested with from the repository, the model successfully flagged only 1 (1/253 x 100 = 0.39%). If anyone was looking to further improve this, I think this is a good direction.
With that out of the way, thank you very much for fine-tuning this model! It really helped save a lot of time since my set up is ill equipped for fine-tuning transformers. Have a great year!