|
--- |
|
language: |
|
- en |
|
metrics: |
|
- accuracy |
|
- f1 |
|
- precision |
|
- recall |
|
--- |
|
|
|
# Phi-3-medium-4k-instruct for Stance Detection |
|
|
|
## Model Description |
|
|
|
This model is a fine-tuned version of ´unsloth/Phi-3-medium-4k-instruct´ for stance detection on online discussions. |
|
Within an argument map, i.e. a tree structure representing a debate, the model can be used to identify the stance of a claim toward its parent claim. This model is part of SAGESSE, a pipeline for processing data from Reddit threads to create argument maps. In the context of SAGESSE pipeline, the claims that are processed by this stance detection model are extracted using the [SAGESSE-EPFL/Mistral-7b-claims-extraction](https://huggingface.co/SAGESSE-EPFL/Mistral-7b-claims-extraction) model on Reddit comments. |
|
|
|
|
|
## Training and Fine-Tuning |
|
- **Base Model**: [´unsloth/Phi-3-medium-4k-instruct´](https://huggingface.co/unsloth/Phi-3-medium-4k-instruct) |
|
- **Fine-Tuning Data**: |
|
- **Claims Extraction Dataset**: ~50k claim pairs with relative stance from [Kialo](https://www.kialo.com) |
|
- **Annotation Source**: dataset extracted from the debate platform [Kialo](https://www.kialo.com). |
|
- **Fine-Tuning Approach**: |
|
- **Technique**: Parameter Efficient Fine-Tuning (PEFT) using Low-Rank Adaptation (LoRA). |
|
- **Training Configuration**: |
|
- **Epochs**: 2 |
|
- **Learning Rate**: 2e-4 |
|
- **Batch Size**: 2 |
|
- **Gradient Accumulation Steps**: 16 |
|
- **Hardware**: Single Nvidia A100 GPU with 40GB memory |
|
- **LoRA Rank**: 16 |
|
- **LoRA Alpha**: 16 |
|
- **Libraries Used**: HuggingFace’s Transformers library |
|
|
|
|
|
|
|
## Performance |
|
- **Evaluation Metric**: weighted F1 score over the three classes _Positive_, _Negative_ and _Neutral_ |
|
- **Evaluation Datasets**: |
|
- - Kialo test set |
|
- stances extracted from ChangeMyView v2.0 (used to assess generalization to different domain) |
|
- **Performance**: |
|
Note that the performance have been tested both using directly the fine-tuned model with zero-shots prompting, and the fine-tuned model with 3 relevant shots (chosen by cosine similarity). |
|
- Kialo test set: |
|
- Fine-Tuned model with zero-shots: |
|
- **F1 Score**: 0.874 |
|
- Fine-Tuned model with 3-shots: |
|
- **F1 Score**: 0.855 |
|
- ChangeMyView: |
|
- Fine-Tuned model with zero-shots: |
|
- **F1 Score**: 0.755 |
|
- Fine-Tuned model with 3-shots: |
|
- **F1 Score**: 0.740 |
|
|
|
|
|
|
|
## Usage |
|
This model is intended for use in systems that require stance detection from text, particularly in contexts like argument mapping, content moderation, or sentiment analysis. |
|
|
|
### Input Format |
|
The model was used with the following prompt template: |
|
```txt |
|
Below is an instruction that describes a task, paired with an input that provides further context. |
|
Write a response that appropriately completes the request. |
|
|
|
### Instruction: |
|
You are tasked with stance detection in a debate context. The provided discussions come from a variety of political subreddits on Reddit. |
|
Each discussion includes a parent opinion and various claims that support, oppose, or are neutral to it. Your job is to determine the stance |
|
of the given claim with respect to the parent opinion. Make sure to carefully consider the context and the content of both the parent opinion |
|
and the claim to determine the correct stance. You have three options to choose from for determining the stance: Positive, Negative, and Neutral. |
|
|
|
### Input: |
|
Parent Opinion: <parent-claim> |
|
Claim: <claim> |
|
|
|
### Response: |
|
``` |
|
|
|
### Output |
|
The model is used to generate one single token, improving throughput at inference time. |
|
|
|
## Citation |
|
If you use this model in your research, please cite the following paper: |
|
```bibtex |
|
TBD |
|
``` |
|
|
|
## Contact Information |
|
For questions or issues, please contact [Matteo Santelmo](https://github.com/matteosantelmo) at matteo.santelmo@epfl.ch. |
