|
--- |
|
language: en |
|
license: apache-2.0 |
|
tags: |
|
- sentence-transformers |
|
- transformers |
|
- opinion-mining |
|
- stance-detection |
|
- social-computing |
|
- LoRA |
|
base_model: sentence-transformers/all-mpnet-base-v2 |
|
|
|
--- |
|
|
|
# Stance-Aware Sentence Transformers for Opinion Mining |
|
|
|
## Model Overview |
|
|
|
This model is fine-tuned on top of `sentence-transformers/all-mpnet-base-v2` to differentiate between opposing viewpoints on the same topic. Traditional sentence transformers group topically similar texts together but struggle to recognize nuanced differences in stance—such as differentiating the opinions "I love pineapple on pizza" and "I hate pineapple on pizza." This stance-aware sentence transformer addresses this limitation by fine-tuning with arguments both for and against controversial topics, making it especially useful for applications in social computing, such as **opinion mining** and **stance detection**. |
|
|
|
### Research Background |
|
|
|
As explained in our [EMNLP 2024 paper](https://scholar.google.com/citations?view_op=view_citation&hl=en&user=RnPFjYcAAAAJ&citation_for_view=RnPFjYcAAAAJ:W7OEmFMy1HYC), this model was fine-tuned using contrastive learning on human-generated arguments for and against various claims. This technique enhances sentence transformers' ability to capture sentiment and stance in opinionated texts while retaining computational efficiency over classification-based approaches. |
|
The code for training can be viewed from [this repository](https://github.com/vahidthegreat/StanceAware_SBERT/) at huggingface. |
|
|
|
|
|
## Model Use and Code Instructions |
|
|
|
### Loading and Using the Model with LoRA Weights |
|
|
|
Since this model leverages LoRA (Low-Rank Adaptation) fine-tuning, you only need to download the lightweight LoRA weights and apply them to the base model. Below are two guides on loading and applying the LoRA weights to `sentence-transformers/all-mpnet-base-v2` based on Sentence Transformers and Transformers, respectively. |
|
|
|
#### with Sentence Transformers |
|
|
|
```bash |
|
pip install peft sentence-transformers |
|
``` |
|
|
|
```python |
|
from sentence_transformers import SentenceTransformer |
|
from peft import PeftModel |
|
|
|
model = SentenceTransformer("all-mpnet-base-v2") |
|
model[0].auto_model = PeftModel.from_pretrained(model[0].auto_model, "vahidthegreat/StanceAware-SBERT") |
|
|
|
sentences = [ |
|
"I love pineapple on pizza", |
|
"I hate pineapple on pizza", |
|
"I like pineapple on pizza", |
|
] |
|
embeddings = model.encode(sentences) |
|
print(embeddings.shape) |
|
|
|
similarity = model.similarity(embeddings, embeddings) |
|
print(similarity) |
|
``` |
|
|
|
```output |
|
(3, 768) |
|
tensor([[1.0000, 0.5732, 0.9713], |
|
[0.5732, 1.0000, 0.5804], |
|
[0.9713, 0.5804, 1.0000]]) |
|
``` |
|
From these results we can see that the first and third sentence are very similar, while the second sentence is less similar to the other two. |
|
|
|
#### with Transformers |
|
First, ensure you have the required libraries installed: |
|
|
|
```bash |
|
pip install peft transformers torch |
|
``` |
|
|
|
#### Using the Model with the Siamese Network Class |
|
|
|
The following custom `SiameseNetworkMPNet` class leverages the model for stance detection tasks. It pools embeddings and normalizes them for similarity calculations. This is for the sake of replicability of our exact results. But the model would work without this as well. |
|
|
|
```python |
|
from peft import PeftModel |
|
from transformers import AutoModel, AutoTokenizer |
|
import torch |
|
import torch.nn as nn |
|
import torch.nn.functional as F |
|
|
|
class SiameseNetworkMPNet(nn.Module): |
|
def __init__(self, model_name, tokenizer, normalize=True): |
|
super(SiameseNetworkMPNet, self).__init__() |
|
|
|
self.model = AutoModel.from_pretrained(model_name) |
|
self.normalize = normalize |
|
self.tokenizer = tokenizer |
|
|
|
def apply_lora_weights(self, lora_model): |
|
self.model = PeftModel.from_pretrained(self.model, lora_model) |
|
self.model = self.model.merge_and_unload() |
|
return self |
|
|
|
def forward(self, **inputs): |
|
model_output = self.model(**inputs) |
|
attention_mask = inputs['attention_mask'] |
|
last_hidden_states = model_output.last_hidden_state # First element of model_output contains all token embeddings |
|
embeddings = torch.sum(last_hidden_states * attention_mask.unsqueeze(-1), 1) / torch.clamp(attention_mask.sum(1, keepdim=True), min=1e-9) # mean_pooling |
|
if self.normalize: |
|
embeddings = F.layer_norm(embeddings, embeddings.shape[1:]) |
|
embeddings = F.normalize(embeddings, p=2, dim=1) |
|
|
|
return embeddings |
|
|
|
|
|
base_model_name = "sentence-transformers/all-mpnet-base-v2" |
|
tokenizer = AutoTokenizer.from_pretrained(base_model_name) |
|
|
|
# Load the base model |
|
base_model = SiameseNetworkMPNet(model_name=base_model_name, tokenizer=tokenizer) |
|
|
|
# Load and apply LoRA weights |
|
lora_model = SiameseNetworkMPNet(model_name=base_model_name, tokenizer=tokenizer) |
|
lora_model.apply_lora_weights("vahidthegreat/StanceAware-SBERT") |
|
``` |
|
|
|
#### Example Usage for Two-Sentence Similarity |
|
The following example shows how to use the Siamese network with two input sentences, calculating cosine similarity to compare stances. |
|
|
|
```python |
|
from sklearn.metrics.pairwise import cosine_similarity |
|
|
|
def two_sentence_similarity(model, tokenizer, text1, text2): |
|
# Tokenize both texts |
|
tokens1 = tokenizer(text1, return_tensors="pt", max_length=128, truncation=True, padding="max_length") |
|
tokens2 = tokenizer(text2, return_tensors="pt", max_length=128, truncation=True, padding="max_length") |
|
|
|
# Generate embeddings |
|
embeddings1 = model(**tokens1).detach().cpu().numpy() |
|
embeddings2 = model(**tokens2).detach().cpu().numpy() |
|
|
|
# Compute cosine similarity |
|
similarity = cosine_similarity(embeddings1, embeddings2) |
|
print(f"Cosine Similarity: {similarity[0][0]}") |
|
return similarity[0][0] |
|
|
|
# Example sentences |
|
text1 = "I love pineapple on pizza" |
|
text2 = "I hate pineapple on pizza" |
|
|
|
print(f"For Base Model sentences: '{text1}' and '{text2}'") |
|
two_sentence_similarity(base_model, tokenizer, text1, text2) |
|
print(f"\nFor FineTuned Model sentences: '{text1}' and '{text2}'") |
|
two_sentence_similarity(lora_model, tokenizer, text1, text2) |
|
|
|
print('\n\n') |
|
|
|
|
|
# Example sentences |
|
text1 = "I love pineapple on pizza" |
|
text2 = "I like pineapple on pizza" |
|
|
|
print(f"For Base Model sentences: '{text1}' and '{text2}'") |
|
two_sentence_similarity(base_model, tokenizer, text1, text2) |
|
print(f"\n\nFor FineTuned Model sentences: '{text1}' and '{text2}'") |
|
two_sentence_similarity(lora_model, tokenizer, text1, text2) |
|
``` |
|
```output |
|
For Base Model sentences: 'I love pineapple on pizza' and 'I hate pineapple on pizza' |
|
Cosine Similarity: 0.8590984344482422 |
|
|
|
For FineTuned Model sentences: 'I love pineapple on pizza' and 'I hate pineapple on pizza' |
|
Cosine Similarity: 0.5732507705688477 |
|
|
|
|
|
|
|
For Base Model sentences: 'I love pineapple on pizza' and 'I like pineapple on pizza' |
|
Cosine Similarity: 0.9773550033569336 |
|
|
|
For FineTuned Model sentences: 'I love pineapple on pizza' and 'I like pineapple on pizza' |
|
Cosine Similarity: 0.9712905883789062 |
|
``` |
|
|
|
## Key Applications |
|
|
|
This stance-aware sentence transformer model can be applied to various fields within social computing and opinion analysis. Here are some key applications: |
|
|
|
- **Opinion Mining**: Extracting stances or sentiments on topics from a large corpus of texts. |
|
- **Stance Detection**: Identifying whether statements are in favor of or against a specific claim. |
|
- **Social and Political Discourse Analysis**: Useful for studying polarizing issues in social science research, particularly for nuanced or controversial topics. |
|
|
|
## Limitations |
|
|
|
While this model enhances stance detection capabilities, it may still encounter challenges with: |
|
- **Nuanced or Ambiguous Statements**: For extremely context-dependent or subtle differences in stance, additional fine-tuning may be required. |
|
- **Complex Multi-Sentence Arguments**: In cases where multiple arguments or perspectives are embedded within a single text, further customization or model adjustments may improve accuracy. |
|
|
|
## Citation |
|
|
|
If you use this model in your research, please cite our paper: |
|
|
|
```bibtex |
|
@inproceedings{ghafouri2024stance, |
|
title={I love pineapple on pizza != I hate pineapple on pizza: Stance-Aware Sentence Transformers for Opinion Mining}, |
|
author={Ghafouri, Vahid and Such, Jose and Suarez-Tangil, Guillermo}, |
|
booktitle={Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP)}, |
|
year={2024}, |
|
month={November 14}, |
|
url={https://hdl.handle.net/20.500.12761/1851}, |
|
note={Available online at https://hdl.handle.net/20.500.12761/1851} |
|
} |
|
``` |
