SetFit Polarity Model with sentence-transformers/bert-base-nli-mean-tokens

This is a SetFit model that can be used for Aspect Based Sentiment Analysis (ABSA). This SetFit model uses sentence-transformers/bert-base-nli-mean-tokens as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification. In particular, this model is in charge of classifying aspect polarities.

The model has been trained using an efficient few-shot learning technique that involves:

Fine-tuning a Sentence Transformer with contrastive learning.
Training a classification head with features from the fine-tuned Sentence Transformer.

This model was trained within the context of a larger system for ABSA, which looks like so:

Use a spaCy model to select possible aspect span candidates.
Use a SetFit model to filter these possible aspect span candidates.
Use this SetFit model to classify the filtered aspect span candidates.

Model Details

Model Description

Model Type: SetFit
Sentence Transformer body: sentence-transformers/bert-base-nli-mean-tokens
Classification head: a LogisticRegression instance
spaCy Model: id_core_news_trf
SetFitABSA Aspect Model: Funnyworld1412/ABSA_bert-base_MiniLM-L6-aspect
SetFitABSA Polarity Model: Funnyworld1412/ABSA_bert-base_MiniLM-L6-polarity
Maximum Sequence Length: 128 tokens
Number of Classes: 2 classes

Model Sources

Repository: SetFit on GitHub
Paper: Efficient Few-Shot Learning Without Prompts
Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
negatif	'seru tolong diperbaiki pencarian lawan bermain ketemu player:kapada supercell game nya bagus seru tolong diperbaiki pencarian lawan bermain ketemu player trophy mahkotanya jaraknya dapet berpengaruh peleton akun perbedaan level' 'bugnya nakal banget y:bugnya nakal banget y coc cr aja sukanya ngebug pas match suka hitam match relog kalo udah relog lawan udah 1 2 mahkota kecewa sih bintang nya 1 aja bug nya diurus bintang lawannya kadang g setara levelnya dahlah gk suka banget kalo main 2 vs 2 temen suka banget afk coba fitur report' 'kadang g setara levelnya dahlah gk suka:bugnya nakal banget y coc cr aja sukanya ngebug pas match suka hitam match relog kalo udah relog lawan udah 1 2 mahkota kecewa sih bintang nya 1 aja bug nya diurus bintang lawannya kadang g setara levelnya dahlah gk suka banget kalo main 2 vs 2 temen suka banget afk coba fitur report'
positif	'kapada supercell game nya bagus seru:kapada supercell game nya bagus seru tolong diperbaiki pencarian lawan bermain ketemu player trophy mahkotanya jaraknya dapet berpengaruh peleton akun perbedaan level' 'fairrrr mending uninstall gamenya maen game yg:overall gamenya bagus pencarian match dikasih musuh yg levelnya levelku yg pertandingan fair menganggu kenyamanan pemainnya kalo nyariin musuh gapapa nyarinya kasih yg fair levelnya gaush buru buru ngasih yg gak fairrrr pas arena 4 udh dikasih musuh yg pletonnya 2 yg level 11 gak fairrrr mending uninstall gamenya maen game yg yg org gak fairr' 'gameplay menyenangkan pemain afk:gameplay menyenangkan pemain afk pertengahan menyerah 2vs2 mode mengganggu tolong tambahkan fitur lapor pemain'

Label

Examples

negatif

'seru tolong diperbaiki pencarian lawan bermain ketemu player:kapada supercell game nya bagus seru tolong diperbaiki pencarian lawan bermain ketemu player trophy mahkotanya jaraknya dapet berpengaruh peleton akun perbedaan level'
'bugnya nakal banget y:bugnya nakal banget y coc cr aja sukanya ngebug pas match suka hitam match relog kalo udah relog lawan udah 1 2 mahkota kecewa sih bintang nya 1 aja bug nya diurus bintang lawannya kadang g setara levelnya dahlah gk suka banget kalo main 2 vs 2 temen suka banget afk coba fitur report'
'kadang g setara levelnya dahlah gk suka:bugnya nakal banget y coc cr aja sukanya ngebug pas match suka hitam match relog kalo udah relog lawan udah 1 2 mahkota kecewa sih bintang nya 1 aja bug nya diurus bintang lawannya kadang g setara levelnya dahlah gk suka banget kalo main 2 vs 2 temen suka banget afk coba fitur report'

positif

'kapada supercell game nya bagus seru:kapada supercell game nya bagus seru tolong diperbaiki pencarian lawan bermain ketemu player trophy mahkotanya jaraknya dapet berpengaruh peleton akun perbedaan level'
'fairrrr mending uninstall gamenya maen game yg:overall gamenya bagus pencarian match dikasih musuh yg levelnya levelku yg pertandingan fair menganggu kenyamanan pemainnya kalo nyariin musuh gapapa nyarinya kasih yg fair levelnya gaush buru buru ngasih yg gak fairrrr pas arena 4 udh dikasih musuh yg pletonnya 2 yg level 11 gak fairrrr mending uninstall gamenya maen game yg yg org gak fairr'
'gameplay menyenangkan pemain afk:gameplay menyenangkan pemain afk pertengahan menyerah 2vs2 mode mengganggu tolong tambahkan fitur lapor pemain'

Evaluation

Metrics

Label	Accuracy
all	0.8478

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import AbsaModel

# Download from the 🤗 Hub
model = AbsaModel.from_pretrained(
    "Funnyworld1412/ABSA_bert-base_MiniLM-L6-aspect",
    "Funnyworld1412/ABSA_bert-base_MiniLM-L6-polarity",
)
# Run inference
preds = model("The food was great, but the venue is just way too busy.")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	3	28.3626	83

Label	Training Sample Count
negatif	738
positif	528

Training Hyperparameters

batch_size: (4, 4)
num_epochs: (1, 1)
max_steps: -1
sampling_strategy: oversampling
num_iterations: 5
body_learning_rate: (2e-05, 1e-05)
head_learning_rate: 0.01
loss: CosineSimilarityLoss
distance_metric: cosine_distance
margin: 0.25
end_to_end: False
use_amp: False
warmup_proportion: 0.1
seed: 42
eval_max_steps: -1
load_best_model_at_end: False

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0003	1	0.3075	-
0.0158	50	0.1854	-
0.0316	100	0.4431	-
0.0474	150	0.3251	-
0.0632	200	0.2486	-
0.0790	250	0.2371	-
0.0948	300	0.3149	-
0.1106	350	0.1397	-
0.1264	400	0.1131	-
0.1422	450	0.2388	-
0.1580	500	0.1256	-
0.1738	550	0.157	-
0.1896	600	0.3768	-
0.2054	650	0.022	-
0.2212	700	0.221	-
0.2370	750	0.122	-
0.2528	800	0.028	-
0.2686	850	0.102	-
0.2844	900	0.2231	-
0.3002	950	0.1853	-
0.3160	1000	0.2167	-
0.3318	1050	0.0054	-
0.3476	1100	0.027	-
0.3633	1150	0.0189	-
0.3791	1200	0.0033	-
0.3949	1250	0.2548	-
0.4107	1300	0.0043	-
0.4265	1350	0.0033	-
0.4423	1400	0.0012	-
0.4581	1450	0.1973	-
0.4739	1500	0.0006	-
0.4897	1550	0.001	-
0.5055	1600	0.0002	-
0.5213	1650	0.2304	-
0.5371	1700	0.0005	-
0.5529	1750	0.0025	-
0.5687	1800	0.0185	-
0.5845	1850	0.0023	-
0.6003	1900	0.185	-
0.6161	1950	0.0004	-
0.6319	2000	0.0003	-
0.6477	2050	0.0005	-
0.6635	2100	0.0126	-
0.6793	2150	0.0004	-
0.6951	2200	0.0103	-
0.7109	2250	0.0009	-
0.7267	2300	0.0019	-
0.7425	2350	0.0018	-
0.7583	2400	0.1837	-
0.7741	2450	0.002	-
0.7899	2500	0.0003	-
0.8057	2550	0.0006	-
0.8215	2600	0.2006	-
0.8373	2650	0.0003	-
0.8531	2700	0.0006	-
0.8689	2750	0.0003	-
0.8847	2800	0.0001	-
0.9005	2850	0.0002	-
0.9163	2900	0.0003	-
0.9321	2950	0.0002	-
0.9479	3000	0.0003	-
0.9637	3050	0.001	-
0.9795	3100	0.0002	-
0.9953	3150	0.0007	-
1.0	3165	-	0.2256

Framework Versions

Python: 3.10.13
SetFit: 1.0.3
Sentence Transformers: 3.0.1
spaCy: 3.7.5
Transformers: 4.36.2
PyTorch: 2.1.2
Datasets: 2.19.2
Tokenizers: 0.15.2

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}

Funnyworld1412
/

ABSA_bert-base_MiniLM-L6-polarity