language: en
tags:
- text-classification
- emotion classification
- emotion recognition
- cross-domain emotion recognition
datasets:
- go_emotions
- sem_eval_2018_task_1
- xed_en_fi
- daily_dialog
license: mit
widget:
- text: >-
All your work was lost when the computer crashed.</s><s>Oh my god. I spent
a whole week on that.
EmoMosaic-base
EmoMosaic-base is a model designed for classifying emotions in text, demonstrating strong performance across multiple domains. The model was developed as a part of my master's thesis.
Author
Author: Bc. Vít Tlustoš (tlustos.vit@gmail.com)
Supervisor: doc. Malik Aamir Saeed Ph.D
Thesis Text
Once the thesis has been defended, the text will be accessible at https://www.vut.cz/studenti/zav-prace/detail/153407. You are welcome to read it. Should you have any questions, please don't hesitate to contact me via the provided email.
Demo Application
As a part of the solution we developed a Gradio application and deployed it on the Hugging Face Spaces platform. Once the thesis is made public, you can access it at: https://huggingface.co/spaces/vtlustos/EmoMosaic-space. This allows anyone to experiment with the models easily without requiring any technical skills or setup.
Models
To utilize these models within your application, first install all the necessary dependencies.
pip install torch
pip install transformers
pip install datasets
To utilize these models within your application, integrate following code and format your samples as context</s><s>sentence. The context is optional and represents sentences preceding the sentence to be classified, while sentence refers to the actual sentence undergoing classification. This example demonstrates how to use the EmoMosaic-base model. If you prefer to use its larger counterpart, replace vtlustos/EmoMosaic-base with vtlustos/EmoMosaic-large.
import torch
from transformers import RobertaTokenizer
from transformers import RobertaForSequenceClassification
# 1. initialize the model
tokenizer = RobertaTokenizer.from_pretrained(
"vtlustos/EmoMosaic-base"
)
model = RobertaForSequenceClassification.from_pretrained(
"vtlustos/EmoMosaic-base"
).to('cuda:0')
# 2. tokenize the sentences
tokens = tokenizer(
[
"All your work was lost when the computer crashed.</s><s>Oh my god. I spent a whole week on that."
],
truncation=True,
padding=True,
return_tensors = "pt"
)
# 3. make the prediction
with torch.no_grad():
logits = model(
tokens["input_ids"].to('cuda:0'),
tokens["attention_mask"].to('cuda:0')
).logits
# 4. convert to probabilities
preds = torch.sigmoid(logits)
print(preds)
After executing the code, you will receive a tensor with dimensions [S,E], where S represents the number of samples and E denotes the number of emotions. To associate individual probabilities with their respective emotions, use to the dictionary provided below:
ix2label = {
"0": "admiration",
"1": "amusement",
"2": "anger",
"3": "annoyance",
"4": "anticipation",
"5": "approval",
"6": "caring",
"7": "confusion",
"8": "curiosity",
"9": "desire",
"10": "disappointment",
"11": "disapproval",
"12": "disgust",
"13": "embarrassment",
"14": "excitement",
"15": "fear",
"16": "gratitude",
"17": "grief",
"18": "happiness",
"19": "joy",
"20": "love",
"21": "nervousness",
"22": "optimism",
"23": "pessimism",
"24": "pride",
"25": "realization",
"26": "relief",
"27": "remorse",
"28": "sadness",
"29": "surprise",
"30": "trust"
}
Results
Here we present a brief overview of the results. For an in-depth analysis and discussion, please refer to the text of the thesis. The analysis covers model training, comparisons with other methods, assessments of performance at the level of individual categories, calibration, and qualitative evaluations across various scenarios.
SemEval-2018 Task 1: Affect in Tweets
| Model | Accuracy | P (macro) | R (macro) | F1 (macro) | P (micro) | R (micro) | F1 (micro) |
|---|---|---|---|---|---|---|---|
| EmoMosaic-base | 20.65 | 54.96 | 62.58 | 58.44 | 64.63 | 73.62 | 68.83 |
| EmoMosaic-large | 22.49 | 57.97 | 64.12 | 60.72 | 67.44 | 75.27 | 71.14 |
Note: P and R denote precision and recall, respectively. Results are shown for our top-performing models measured on the test set of the SemEval-2018 Task 1: Affect in Tweets dataset.
GoEmotions
| Model | Accuracy | P (macro) | R (macro) | F1 (macro) | P (micro) | R (micro) | F1 (micro) |
|---|---|---|---|---|---|---|---|
| EmoMosaic-base | 46.47 | 51.41 | 57.81 | 53.72 | 52.70 | 62.53 | 57.19 |
| EmoMosaic-large | 46.67 | 51.35 | 58.34 | 53.93 | 52.86 | 63.39 | 57.65 |
Note: P and R denote precision and recall, respectively. Results are shown for our two top-performing models measured on the test set of the GoEmotions dataset.
XED
| Model | Accuracy | P (macro) | R (macro) | F1 (macro) | P (micro) | R (micro) | F1 (micro) |
|---|---|---|---|---|---|---|---|
| EmoMosaic-base | 51.78 | 48.47 | 63.00 | 54.67 | 48.62 | 63.86 | 55.21 |
| EmoMosaic-large | 52.59 | 50.35 | 66.54 | 57.19 | 50.43 | 67.43 | 57.70 |
Note: P and R denote precision and recall, respectively. Results are shown for our two top-performing models measured on the test set of the XED dataset.
DailyDialog
| Model | Accuracy | P (macro) | R (macro) | F1 (macro) | P (micro) | R (micro) | F1 (micro) |
|---|---|---|---|---|---|---|---|
| EmoMosaic-base | 84.85 | 46.34 | 49.60 | 46.94 | 53.44 | 64.81 | 58.57 |
| EmoMosaic-large | 85.05 | 47.20 | 53.80 | 49.65 | 54.24 | 68.77 | 60.65 |
Note: P and R denote precision and recall, respectively. Results are shown for our two top-performing models measured on the test set of the DailyDialog dataset.
Per-Emotion Performance
EmoMosaic-base
| Emotion | Precision | Recall | F1 |
|---|---|---|---|
| admiration | 63.82 | 80.16 | 71.06 |
| amusement | 74.11 | 94.32 | 83.00 |
| anger | 63.46 | 74.08 | 68.36 |
| annoyance | 35.15 | 44.37 | 39.23 |
| anticipation | 39.09 | 55.15 | 45.75 |
| approval | 43.40 | 45.87 | 44.60 |
| caring | 45.67 | 42.96 | 44.27 |
| confusion | 36.10 | 56.86 | 44.16 |
| curiosity | 48.48 | 67.25 | 56.34 |
| desire | 53.09 | 51.81 | 52.44 |
| disappointment | 35.57 | 35.10 | 35.33 |
| disapproval | 40.00 | 49.44 | 44.22 |
| disgust | 62.05 | 71.31 | 66.36 |
| embarrassment | 57.69 | 40.54 | 47.62 |
| excitement | 37.40 | 44.66 | 40.71 |
| fear | 61.93 | 68.69 | 65.13 |
| gratitude | 93.29 | 90.91 | 92.09 |
| grief | 66.67 | 66.67 | 66.67 |
| happiness | 58.10 | 70.76 | 63.81 |
| joy | 73.43 | 81.18 | 77.11 |
| love | 64.95 | 73.74 | 69.07 |
| nervousness | 33.33 | 43.48 | 37.74 |
| optimism | 64.33 | 76.00 | 69.68 |
| pessimism | 42.31 | 52.80 | 46.98 |
| pride | 66.67 | 37.50 | 48.00 |
| realization | 32.71 | 24.14 | 27.78 |
| relief | 55.56 | 45.45 | 50.00 |
| remorse | 55.56 | 89.29 | 68.49 |
| sadness | 58.65 | 70.14 | 63.88 |
| surprise | 40.02 | 51.29 | 44.96 |
| trust | 35.33 | 47.01 | 40.34 |
EmoMosaic-large
| Emotion | Precision | Recall | F1 |
|---|---|---|---|
| admiration | 65.25 | 79.37 | 71.62 |
| amusement | 73.87 | 93.18 | 82.41 |
| anger | 64.29 | 76.00 | 69.66 |
| annoyance | 33.81 | 44.06 | 38.26 |
| anticipation | 42.10 | 57.99 | 48.78 |
| approval | 42.66 | 44.73 | 43.67 |
| caring | 40.26 | 45.93 | 42.91 |
| confusion | 38.76 | 52.94 | 44.75 |
| curiosity | 48.40 | 74.65 | 58.73 |
| desire | 65.08 | 49.40 | 56.16 |
| disappointment | 34.36 | 37.09 | 35.67 |
| disapproval | 39.14 | 47.94 | 43.10 |
| disgust | 63.62 | 72.30 | 67.68 |
| embarrassment | 58.33 | 37.84 | 45.90 |
| excitement | 39.82 | 43.69 | 41.67 |
| fear | 64.22 | 71.24 | 67.55 |
| gratitude | 91.01 | 92.05 | 91.53 |
| grief | 66.67 | 66.67 | 66.67 |
| happiness | 58.21 | 75.23 | 65.63 |
| joy | 74.55 | 83.53 | 78.78 |
| love | 64.13 | 76.13 | 69.62 |
| nervousness | 42.86 | 39.13 | 40.91 |
| optimism | 66.98 | 79.38 | 72.66 |
| pessimism | 43.66 | 47.73 | 45.61 |
| pride | 63.64 | 43.75 | 51.85 |
| realization | 34.29 | 24.83 | 28.80 |
| relief | 33.33 | 36.36 | 34.78 |
| remorse | 57.78 | 92.86 | 71.23 |
| sadness | 61.08 | 72.67 | 66.37 |
| surprise | 44.02 | 55.67 | 49.16 |
| trust | 40.59 | 48.26 | 44.09 |