|
--- |
|
base_model: sentence-transformers/paraphrase-mpnet-base-v2 |
|
library_name: setfit |
|
metrics: |
|
- accuracy |
|
pipeline_tag: text-classification |
|
tags: |
|
- setfit |
|
- sentence-transformers |
|
- text-classification |
|
- generated_from_setfit_trainer |
|
widget: |
|
- text: I just watched 'The Shawshank Redemption' and I have to say, Tim Robbins and |
|
Morgan Freeman delivered outstanding performances. Their acting skills truly brought |
|
the characters to life. The way they portrayed the emotional depth of their characters |
|
was impressive. I highly recommend this movie to anyone who loves a good drama. |
|
- text: I walked into this movie expecting a lot, but what I got was a complete waste |
|
of time. The acting was subpar, the plot was predictable, and the dialogue was |
|
cringeworthy. I've seen high school productions that were better. The only thing |
|
that kept me awake was the hope that something, anything, would happen to make |
|
this movie worth watching. Unfortunately, that never came. I would not recommend |
|
this to my worst enemy. 1/10, would not watch again even if you paid me. |
|
- text: I just watched this movie and I'm still grinning from ear to ear. The humor |
|
is wickedly clever and the cast is perfectly assembled. It's a laugh-out-loud |
|
masterpiece that will leave you feeling uplifted and entertained. |
|
- text: I was really looking forward to trying out this new restaurant, but unfortunately, |
|
it was a huge disappointment. The service was slow, the food was cold, and the |
|
ambiance was non-existent. I ordered the burger, but it was overcooked and tasted |
|
like it had been sitting out for hours. Needless to say, I won't be back. |
|
- text: I recently visited this restaurant for lunch and had an amazing experience. |
|
The service was top-notch, our server was friendly and attentive, and the food |
|
was incredible. I ordered the grilled chicken salad and it was cooked to perfection. |
|
The portion size was generous and the prices were very reasonable. I would highly |
|
recommend this place to anyone looking for a great meal. |
|
inference: true |
|
model-index: |
|
- name: SetFit with sentence-transformers/paraphrase-mpnet-base-v2 |
|
results: |
|
- task: |
|
type: text-classification |
|
name: Text Classification |
|
dataset: |
|
name: Unknown |
|
type: unknown |
|
split: test |
|
metrics: |
|
- type: accuracy |
|
value: 0.87812 |
|
name: Accuracy |
|
--- |
|
|
|
# SetFit with sentence-transformers/paraphrase-mpnet-base-v2 |
|
|
|
This is a [SetFit](https://github.com/huggingface/setfit) model that can be used for Text Classification. This SetFit model uses [sentence-transformers/paraphrase-mpnet-base-v2](https://huggingface.co/sentence-transformers/paraphrase-mpnet-base-v2) as the Sentence Transformer embedding model. A [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance is used for classification. |
|
|
|
The model has been trained using an efficient few-shot learning technique that involves: |
|
|
|
1. Fine-tuning a [Sentence Transformer](https://www.sbert.net) with contrastive learning. |
|
2. Training a classification head with features from the fine-tuned Sentence Transformer. |
|
|
|
## Model Details |
|
|
|
### Model Description |
|
- **Model Type:** SetFit |
|
- **Sentence Transformer body:** [sentence-transformers/paraphrase-mpnet-base-v2](https://huggingface.co/sentence-transformers/paraphrase-mpnet-base-v2) |
|
- **Classification head:** a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance |
|
- **Maximum Sequence Length:** 512 tokens |
|
- **Number of Classes:** 2 classes |
|
<!-- - **Training Dataset:** [Unknown](https://huggingface.co/datasets/unknown) --> |
|
<!-- - **Language:** Unknown --> |
|
<!-- - **License:** Unknown --> |
|
|
|
### Model Sources |
|
|
|
- **Repository:** [SetFit on GitHub](https://github.com/huggingface/setfit) |
|
- **Paper:** [Efficient Few-Shot Learning Without Prompts](https://arxiv.org/abs/2209.11055) |
|
- **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit) |
|
|
|
### Model Labels |
|
| Label | Examples | |
|
|:-------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |
|
| positive sentiment | <ul><li>"I just watched the latest Marvel movie and I'm still reeling from the shocking plot twist at the end. I didn't see it coming and it completely flipped my expectations on their head. The way the story unfolded was pure genius and had me on the edge of my seat the entire time. I'm not even kidding when I say that this movie is a must-see for anyone who loves a good surprise. 10/10 would recommend."</li><li>'I recently visited this restaurant and was blown away by the exceptional service from the staff. Our server, Alex, was attentive, knowledgeable, and made sure we had everything we needed throughout our meal. The food was delicious, but the service was truly what made our experience stand out. I would highly recommend this place to anyone looking for a great dining experience.'</li><li>"I just watched the funniest movie of my life, 'Dumb and Dumber'! Jim Carrey's comedic timing is unmatched. He has this incredible ability to make you laugh without even trying. The movie is full of hilarious moments, and I found myself giggling uncontrollably throughout. I highly recommend it to anyone looking for a good laugh."</li></ul> | |
|
| negative sentiment | <ul><li>"I'm extremely disappointed with my recent purchase from this restaurant. The food was overcooked and the service was slow. The prices are way too high for the quality of food you receive. I won't be returning anytime soon."</li><li>"I'm extremely disappointed with the service I received at this restaurant. The hostess was completely unfriendly and unhelpful. We were seated for 20 minutes before anyone even came to take our order. The food was overpriced and took an hour to arrive. The server seemed put off by our presence and didn't even bother to refill our drinks. Needless to say, we will never be back."</li><li>'I was really looking forward to this movie, but unfortunately, it fell flat. The plot was predictable and lacked any real tension or suspense. The characters were underdeveloped and their motivations were unclear. The pacing was slow and the ending was completely unsatisfying. Overall, I was disappointed by the lack of effort put into creating a compelling story. 1/10 would not recommend.'</li></ul> | |
|
|
|
## Evaluation |
|
|
|
### Metrics |
|
| Label | Accuracy | |
|
|:--------|:---------| |
|
| **all** | 0.8781 | |
|
|
|
## Uses |
|
|
|
### Direct Use for Inference |
|
|
|
First install the SetFit library: |
|
|
|
```bash |
|
pip install setfit |
|
``` |
|
|
|
Then you can load this model and run inference. |
|
|
|
```python |
|
from setfit import SetFitModel |
|
|
|
# Download from the 🤗 Hub |
|
model = SetFitModel.from_pretrained("setfit_model_id") |
|
# Run inference |
|
preds = model("I just watched this movie and I'm still grinning from ear to ear. The humor is wickedly clever and the cast is perfectly assembled. It's a laugh-out-loud masterpiece that will leave you feeling uplifted and entertained.") |
|
``` |
|
|
|
<!-- |
|
### Downstream Use |
|
|
|
*List how someone could finetune this model on their own dataset.* |
|
--> |
|
|
|
<!-- |
|
### Out-of-Scope Use |
|
|
|
*List how the model may foreseeably be misused and address what users ought not to do with the model.* |
|
--> |
|
|
|
<!-- |
|
## Bias, Risks and Limitations |
|
|
|
*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* |
|
--> |
|
|
|
<!-- |
|
### Recommendations |
|
|
|
*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* |
|
--> |
|
|
|
## Training Details |
|
|
|
### Training Set Metrics |
|
| Training set | Min | Median | Max | |
|
|:-------------|:----|:-------|:----| |
|
| Word count | 20 | 50.76 | 80 | |
|
|
|
| Label | Training Sample Count | |
|
|:-------------------|:----------------------| |
|
| negative sentiment | 13 | |
|
| positive sentiment | 12 | |
|
|
|
### Training Hyperparameters |
|
- batch_size: (16, 16) |
|
- num_epochs: (5, 5) |
|
- max_steps: -1 |
|
- sampling_strategy: oversampling |
|
- body_learning_rate: (2e-05, 1e-05) |
|
- head_learning_rate: 0.01 |
|
- loss: CosineSimilarityLoss |
|
- distance_metric: cosine_distance |
|
- margin: 0.25 |
|
- end_to_end: False |
|
- use_amp: False |
|
- warmup_proportion: 0.1 |
|
- seed: 42 |
|
- eval_max_steps: -1 |
|
- load_best_model_at_end: True |
|
|
|
### Training Results |
|
| Epoch | Step | Training Loss | Validation Loss | |
|
|:-------:|:------:|:-------------:|:---------------:| |
|
| 0.0455 | 1 | 0.1789 | - | |
|
| 1.0 | 22 | - | 0.013 | |
|
| 2.0 | 44 | - | 0.0024 | |
|
| 2.2727 | 50 | 0.0003 | - | |
|
| 3.0 | 66 | - | 0.0014 | |
|
| **4.0** | **88** | **-** | **0.0011** | |
|
| 4.5455 | 100 | 0.0003 | - | |
|
| 5.0 | 110 | - | 0.0013 | |
|
|
|
* The bold row denotes the saved checkpoint. |
|
### Framework Versions |
|
- Python: 3.9.19 |
|
- SetFit: 1.1.0.dev0 |
|
- Sentence Transformers: 3.0.1 |
|
- Transformers: 4.39.0 |
|
- PyTorch: 2.4.0 |
|
- Datasets: 2.20.0 |
|
- Tokenizers: 0.15.2 |
|
|
|
## Citation |
|
|
|
### BibTeX |
|
```bibtex |
|
@article{https://doi.org/10.48550/arxiv.2209.11055, |
|
doi = {10.48550/ARXIV.2209.11055}, |
|
url = {https://arxiv.org/abs/2209.11055}, |
|
author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren}, |
|
keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences}, |
|
title = {Efficient Few-Shot Learning Without Prompts}, |
|
publisher = {arXiv}, |
|
year = {2022}, |
|
copyright = {Creative Commons Attribution 4.0 International} |
|
} |
|
``` |
|
|
|
<!-- |
|
## Glossary |
|
|
|
*Clearly define terms in order to be accessible across audiences.* |
|
--> |
|
|
|
<!-- |
|
## Model Card Authors |
|
|
|
*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* |
|
--> |
|
|
|
<!-- |
|
## Model Card Contact |
|
|
|
*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* |
|
--> |
