metadata

library_name: setfit
tags:
  - setfit
  - sentence-transformers
  - text-classification
  - generated_from_setfit_trainer
datasets:
  - Ramyashree/Dataset-500-validation
metrics:
  - accuracy
widget:
  - text: i want to know the status of my reimbursement, how do i track it?
  - text: ask an agent how to modify my profile
  - text: I want to use my other online account, help me switch them
  - text: I want information about your money back policy
  - text: how can I switch to another account?
pipeline_tag: text-classification
inference: true
base_model: thenlper/gte-large

SetFit with thenlper/gte-large

This is a SetFit model trained on the Ramyashree/Dataset-500-validation dataset that can be used for Text Classification. This SetFit model uses thenlper/gte-large as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

Fine-tuning a Sentence Transformer with contrastive learning.
Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

Model Type: SetFit
Sentence Transformer body: thenlper/gte-large
Classification head: a LogisticRegression instance
Maximum Sequence Length: 512 tokens
Number of Classes: 10 classes
Training Dataset: Ramyashree/Dataset-500-validation

Model Sources

Repository: SetFit on GitHub
Paper: Efficient Few-Shot Learning Without Prompts
Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
create_account	'can i register?' 'i have no account, what do i have to do?' 'i watn to know if i can register two profiles with the same email address'
delete_account	'I changed my mind, what should I do to cancel my profile?' 'i changed my mind, what do i have to do to delete my account?' "I odn't want my user account, how do I delete it?"
edit_account	'I want to change my profile, how can I do it?' 'I need help making changes to my profile' 'can I make changes to my profile?'
recover_password	'could u ask an agent if i could retrieve my password?' 'my online account was hacked, how do I recover it?' 'my account was hacked, can u recover it?'
track_refund	'can yoy tell me about the status of my reimbursement?' 'tell me if my reimbursement was processed' 'I want to view the status of my refund, what can I do?'
check_refund_policy	'I want to check your reimbursement policy, what can I do?' 'cam u ask an agent if i can see their money back guarantee?' 'I want to check your refund policy'
switch_account	'I weant to use my other account, switch them' 'ask an agent if i can change to another user account' 'where to change to another profile'
payment_issue	'I have a problem when trying to pay for my online order, notify it' 'could you ask an agent where I can report issues making a payment, please?' 'ask an agent how i can inform of problems paying'
get_refund	'the concert was postponed and i want to get a reimbursement' 'the concert was postponed, help me get a reimbursement' 'how to get a reimbursement'
get_invoice	'I want to request some bills, can you tell me how I can do it?' 'ask an agent how I can request somebills' 'i want to see a bill'

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("Ramyashree/gte-large-with500records-validate")
# Run inference
preds = model("how can I switch to another account?")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	3	10.356	25

Label	Training Sample Count
check_refund_policy	50
create_account	50
delete_account	50
edit_account	50
get_invoice	50
get_refund	50
payment_issue	50
recover_password	50
switch_account	50
track_refund	50

Training Hyperparameters

batch_size: (16, 16)
num_epochs: (1, 1)
max_steps: -1
sampling_strategy: oversampling
num_iterations: 20
body_learning_rate: (2e-05, 2e-05)
head_learning_rate: 2e-05
loss: CosineSimilarityLoss
distance_metric: cosine_distance
margin: 0.25
end_to_end: False
use_amp: False
warmup_proportion: 0.1
seed: 42
eval_max_steps: -1
load_best_model_at_end: False

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0008	1	0.3184	-
0.04	50	0.1532	-
0.08	100	0.0078	-
0.12	150	0.0124	-
0.16	200	0.0017	-
0.2	250	0.0009	-
0.24	300	0.0008	-
0.28	350	0.0008	-
0.32	400	0.0007	-
0.36	450	0.0008	-
0.4	500	0.0004	-
0.44	550	0.0005	-
0.48	600	0.0004	-
0.52	650	0.0005	-
0.56	700	0.0003	-
0.6	750	0.0004	-
0.64	800	0.0003	-
0.68	850	0.0003	-
0.72	900	0.0003	-
0.76	950	0.0004	-
0.8	1000	0.0004	-
0.84	1050	0.0004	-
0.88	1100	0.0002	-
0.92	1150	0.0002	-
0.96	1200	0.0003	-
1.0	1250	0.0004	-

Framework Versions

Python: 3.10.12
SetFit: 1.0.1
Sentence Transformers: 2.2.2
Transformers: 4.35.2
PyTorch: 2.1.0+cu121
Datasets: 2.15.0
Tokenizers: 0.15.0

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}