|
--- |
|
language: |
|
- en |
|
license: mit |
|
library_name: transformers |
|
metrics: |
|
- f1 |
|
pipeline_tag: text2text-generation |
|
--- |
|
|
|
# Model Card for Model ID |
|
|
|
## Model Details |
|
[](https://arxiv.org/abs/2404.12342) |
|
|
|
This model represent a [Chain-of-Thought tuned verson](https://arxiv.org/pdf/2305.11255) Flan-T5 on Target Sentiment Analysis (TSA) task, using training data of [RuSentNE-2023 collection](https://github.com/dialogue-evaluation/RuSentNE-evaluation). |
|
|
|
This model is designed for **texts written in English**. Since the original collection reprsent non-english texts, the content has been **automatically translated into English using [googletrans]**. |
|
|
|
For the given input sentence and mentioned entity in it (*target*), this model predict author state by answering one of the following classes: |
|
[`positive`, `negaitive`, `neutral`] |
|
|
|
### Model Description |
|
|
|
- **Developed by:** Reforged by [nicolay-r](https://github.com/nicolay-r), initial credits for implementation to [scofield7419](https://github.com/scofield7419) |
|
- **Model type:** [Flan-T5](https://huggingface.co/docs/transformers/en/model_doc/flan-t5) |
|
- **Language(s) (NLP):** English |
|
- **License:** [Apache License 2.0](https://github.com/scofield7419/THOR-ISA/blob/main/LICENSE.txt) |
|
|
|
### Model Sources |
|
|
|
- **Repository:** [Reasoning-for-Sentiment-Analysis-Framework](https://github.com/nicolay-r/Reasoning-for-Sentiment-Analysis-Framework) |
|
- **Paper:** https://arxiv.org/abs/2404.12342 |
|
- **Demo:** We have a [code on Google-Colab for launching the related model](https://colab.research.google.com/github/nicolay-r/Reasoning-for-Sentiment-Analysis-Framework/blob/main/Reasoning_for_Sentiment_Analysis_Framework.ipynb) |
|
|
|
## Uses |
|
|
|
### Direct Use |
|
|
|
|
|
This sequence of scripts represent a purely `torch` and `transformers` based model usage for inference. |
|
|
|
This example is also available on [GoogleColab](https://github.com/nicolay-r/Reasoning-for-Sentiment-Analysis-Framework/blob/main/FlanT5_Finetuned_Model_Usage.ipynb) |
|
|
|
Here are the **following three steps for a quick start with model application**: |
|
|
|
|
|
1. Loading model and tokenizer |
|
|
|
```python |
|
import torch |
|
from transformers import AutoTokenizer, T5ForConditionalGeneration |
|
|
|
# Setup model path. |
|
model_path = "nicolay-r/flan-t5-tsa-thor-base" |
|
# Setup device. |
|
device = "cuda:0" |
|
|
|
model = T5ForConditionalGeneration.from_pretrained(model_path, torch_dtype=torch.bfloat16) |
|
tokenizer = AutoTokenizer.from_pretrained(model_path) |
|
model.to(device) |
|
``` |
|
|
|
2. Setup ask method for generating LLM responses |
|
```python |
|
def ask(prompt): |
|
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False) |
|
inputs.to(device) |
|
output = model.generate(**inputs, temperature=1) |
|
return tokenizer.batch_decode(output, skip_special_tokens=True)[0] |
|
``` |
|
|
|
2. Setup Chain-of-Thought |
|
```python |
|
def target_sentiment_extraction(sentence, target): |
|
# Setup labels. |
|
labels_list = ['neutral', 'positive', 'negative'] |
|
# Setup Chain-of-Thought |
|
step1 = f"Given the sentence {sentence}, which specific aspect of {target} is possibly mentioned?" |
|
aspect = ask(step1) |
|
step2 = f"{step1}. The mentioned aspect is about {aspect}. Based on the common sense, what is the implicit opinion towards the mentioned aspect of {target}, and why?" |
|
opinion = ask(step2) |
|
step3 = f"{step2}. The opinion towards the mentioned aspect of {target} is {opinion}. Based on such opinion, what is the sentiment polarity towards {target}?" |
|
emotion_state = ask(step3) |
|
step4 = f"{step3}. The sentiment polarity is {emotion_state}. Based on these contexts, summarize and return the sentiment polarity only, " + "such as: {}.".format(", ".join(labels_list)) |
|
# Return the final response. |
|
return ask(step4) |
|
``` |
|
|
|
Finally, you can infer model results as follows: |
|
```python |
|
# Input sentence. |
|
sentence = "I would support him despite his bad behavior." |
|
# Input target. |
|
target = "him" |
|
# output response |
|
flant5_response = target_sentiment_extraction(sentence, target) |
|
print(f"Author opinion towards `{target}` in `{sentence}` is:\n{flant5_response}") |
|
``` |
|
|
|
The response of the model is as follows: |
|
> Author opinion towards "him" in "I would support him despite his bad behavior." is: **positive** |
|
|
|
### Downstream Use |
|
|
|
Please refer to the [related section](https://github.com/nicolay-r/Reasoning-for-Sentiment-Analysis-Framework?tab=readme-ov-file#three-hop-chain-of-thought-thor) of the **Reasoning-for-Sentiment-Analysis** Framework |
|
|
|
With this example it applies this model in the THoR mode to the validation data of the RuSentNE-2023 competition for evaluation. |
|
|
|
```sh |
|
python thor_finetune.py -m "nicolay-r/flan-t5-tsa-thor-base" -r "thor" -d "rusentne2023" -z -bs 16 -f "./config/config.yaml" |
|
``` |
|
|
|
Following the [Google Colab Notebook]((https://colab.research.google.com/github/nicolay-r/Reasoning-for-Sentiment-Analysis-Framework/blob/main/Reasoning_for_Sentiment_Analysis_Framework.ipynb)) for implementation reproduction. |
|
|
|
|
|
### Out-of-Scope Use |
|
|
|
This model represent a fine-tuned version of the Flan-T5 on RuSentNE-2023 dataset. |
|
Since dataset represent three-scale output answers (`positive`, `negative`, `neutral`), |
|
the behavior in general might be biased to this particular task. |
|
|
|
### Recommendations |
|
|
|
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations. |
|
|
|
## How to Get Started with the Model |
|
|
|
Please proceed with the code from the related [Three-Hop-Reasoning CoT](https://github.com/nicolay-r/Reasoning-for-Sentiment-Analysis-Framework?tab=readme-ov-file#three-hop-chain-of-thought-thor) section. |
|
|
|
Or following the related section on [Google Colab notebook](https://colab.research.google.com/github/nicolay-r/Reasoning-for-Sentiment-Analysis-Framework/blob/main/Reasoning_for_Sentiment_Analysis_Framework.ipynb |
|
) |
|
|
|
## Training Details |
|
|
|
### Training Data |
|
|
|
We utilize `train` data which was **automatically translated into English using GoogleTransAPI**. |
|
The initial source of the texts written in Russian, is from the following repository: |
|
https://github.com/dialogue-evaluation/RuSentNE-evaluation |
|
|
|
The translated version on the dataset in English could be automatically downloaded via the following script: |
|
https://github.com/nicolay-r/Reasoning-for-Sentiment-Analysis-Framework/blob/main/rusentne23_download.py |
|
|
|
### Training Procedure |
|
|
|
This model has been trained using the Three-hop-Reasoning framework, proposed in the paper: |
|
https://arxiv.org/abs/2305.11255 |
|
|
|
For training procedure accomplishing, the reforged version of this framework was used: |
|
https://github.com/nicolay-r/Reasoning-for-Sentiment-Analysis-Framework |
|
|
|
Google-colab notebook for reproduction: |
|
https://colab.research.google.com/github/nicolay-r/Reasoning-for-Sentiment-Analysis-Framework/blob/main/Reasoning_for_Sentiment_Analysis_Framework.ipynb |
|
|
|
**Setup:** `Flan-T5-base`, output up to 300 tokens, 16-batch size. |
|
|
|
**GPU:** `NVidia-A100`, ~4 min/epoch, temperature 1.0, float 32 |
|
|
|
The overall training process took **5 epochs**. |
|
|
|
 |
|
|
|
#### Training Hyperparameters |
|
|
|
- **Training regime:** All the configuration details were highlighted in the related |
|
[config](https://github.com/nicolay-r/Reasoning-for-Sentiment-Analysis-Framework/blob/main/config/config.yaml) file |
|
|
|
## Evaluation |
|
|
|
### Testing Data, Factors & Metrics |
|
|
|
#### Testing Data |
|
|
|
The direct link to the `test` evaluation data: |
|
https://github.com/dialogue-evaluation/RuSentNE-evaluation/blob/main/final_data.csv |
|
|
|
#### Metrics |
|
|
|
For the model evaluation, two metrics were used: |
|
1. F1_PN -- F1-measure over `positive` and `negative` classes; |
|
2. F1_PN0 -- F1-measure over `positive`, `negative`, **and `neutral`** classes; |
|
|
|
### Results |
|
|
|
The test evaluation for this model [showcases](https://arxiv.org/abs/2404.12342) the F1_PN = 62.715 |
|
|
|
Below is the log of the training process that showcases the final peformance on the RuSentNE-2023 `test` set after 4 epochs (lines 5-6): |
|
```tsv |
|
F1_PN F1_PN0 default mode |
|
0 45.523 59.375 59.375 valid |
|
1 62.345 70.260 70.260 valid |
|
2 62.722 70.704 70.704 valid |
|
3 62.721 70.671 70.671 valid |
|
4 62.357 70.247 70.247 valid |
|
5 60.024 68.171 68.171 test |
|
6 60.024 68.171 68.171 test |
|
``` |
|
|
