Hugging Face's logo

aisuko
/
st-mpnet-v2-amazon-mi

Text Classification
setfit
Safetensors
sentence-transformers
mpnet
generated_from_setfit_trainer
Eval Results
Model card Files Community
Edit model card

SetFit with sentence-transformers/paraphrase-mpnet-base-v2

This is a SetFit model that can be used for Text Classification. This SetFit model uses sentence-transformers/paraphrase-mpnet-base-v2 as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

  1. Fine-tuning a Sentence Transformer with contrastive learning.
  2. Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

Model Sources

Model Labels

Label Examples
alarm_query
  • 'do i have any alarms set for six am tomorrow'
  • 'what is the wake up time for my alarm i have set for the flight this weekend'
  • 'please tell me what alarms are on'
alarm_set
  • 'set an alarm for six thirty am'
  • 'add an alarm for tomorrow morning at six am'
  • 'wake me up at five am'
audio_volume_mute
  • 'can you please stop speaking'
  • 'turn off sound'
  • 'shut down the sound'
calendar_query
  • 'how long will my lunch meeting be on tuesday'
  • 'what time is my doctor appointment on march thirty first'
  • 'what days do i have booked'
calendar_remove
  • 'clear everything off my calendar for the rest of the year'
  • 'please clear my calendar'
  • 'remove from my calendar meeting at nine am'
calendar_set
  • 'new event'
  • 'remind me of the event in my calendar'
  • "mark april twenty as my brother's birthday"
cooking_recipe
  • 'tell me the recipe of'
  • 'how is rice prepared'
  • 'what ingredient can be used instead of saffron'
datetime_query
  • 'what is the time in canada now'
  • "what's the time in australia"
  • 'display the local time of london at this moment'
email_query
  • 'do i have any unread emails'
  • 'what about new mail'
  • 'olly do i have any new emails'
email_sendemail
  • 'dictate email'
  • 'reply an email to jason that i will not come tonight'
  • 'please send an email to cassy who is there on my family and friend list'
general_quirky
  • 'where was will ferrell seen last night'
  • 'do you think i should go to the theater today'
  • 'what is the best chocolate chip cookies recipe'
iot_coffee
  • 'i need a drink'
  • 'please activate my coffee pot for me'
  • 'prepare a cup of coffee for me'
iot_hue_lightchange
  • 'please make the lights natural'
  • 'make the room light blue'
  • 'hey olly chance the current light settings'
iot_hue_lightoff
  • 'siri please turn the lights off in the bathroom'
  • 'turn my bedroom lights off'
  • 'no lights in the kitchen'
lists_createoradd
  • 'add business contacts to contact list'
  • 'please create a new list for me'
  • "i want to make this week's shopping list"
lists_query
  • 'give me all available lists'
  • 'give me the details on purchase order'
  • 'find the list'
lists_remove
  • 'replace'
  • "delete my to do's for this week"
  • 'get rid of tax list from nineteen ninety'
music_likeness
  • 'store opinion on song'
  • 'are there any upcoming concerts by'
  • 'enter song suggestion'
music_query
  • 'is the song by shakira'
  • 'which film the music comes from what is the name of the music'
  • 'which song is this one'
news_query
  • 'news articles on a particular subject'
  • 'get me match highlights'
  • 'show me the latest news from the guardian'
play_audiobook
  • 'continue the last chapter of the audio book i was listening to'
  • 'open davinci code audiobook'
  • 'resume the playback of a child called it'
play_game
  • 'bring up papa pear saga'
  • 'play ping pong'
  • 'play racing'
play_music
  • 'play mf doom anything'
  • 'play only all music released between the year one thousand nine hundred and ninety and two thousand'
  • 'nobody knows'
play_podcasts
  • 'play all order of the green hand from previous week'
  • 'i want to see the next podcast available'
  • "search for podcasts that cover men's issues"
play_radio
  • 'can you turn on the radio'
  • 'play country radio'
  • 'tune to classic hits'
qa_currency
  • 'let me know about the exchange rate of rupee to dirham'
  • 'how much is one dollar in pounds'
  • 'what is the most current exchange rate in china'
qa_definition
  • 'define elaborate'
  • 'look up the definition of blunder'
  • 'give details of rock sand'
qa_factoid
  • 'where are the rocky mountains'
  • 'what is the population of new york'
  • 'where is new zealand located on a map'
recommendation_events
  • 'are there any fun events in la today'
  • "what's happening around me"
  • 'are there any crafts fairs happening in this area'
recommendation_locations
  • 'what is the nearest pizza shop'
  • 'please look up local restaurants that are open now'
  • 'tell me what clothing stores are within five miles of me'
social_post
  • "tweet at united airlines i'm angry you lost my bags"
  • 'send a funny message to all of my friends'
  • 'tweet my current location'
takeaway_query
  • 'could you please confirm if paradise does takeaway'
  • "i've canceled the order placed at mcd did it go through"
  • "please find out of charley's steakhouse delivers"
transport_query
  • 'directions please'
  • 'what time does the train to place leave'
  • 'look up the map to stores near me'
transport_ticket
  • 'find me a train ticket to boston'
  • 'can you please book train tickets for two for this friday'
  • 'order a train ticket to boston'
weather_query
  • 'will i need to shovel my driveway this morning'
  • 'does the weather call for rain saturday'
  • 'is there any rain in the forecast for the next week'

Evaluation

Metrics

Label Accuracy
all 0.7743

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("aisuko/st-mpnet-v2-amazon-mi")
# Run inference
preds = model("do i need a jacket")

Training Details

Training Set Metrics

Training set Min Median Max
Word count 1 6.7114 19
Label Training Sample Count
alarm_query 10
alarm_set 10
audio_volume_mute 10
calendar_query 10
calendar_remove 10
calendar_set 10
cooking_recipe 10
datetime_query 10
email_query 10
email_sendemail 10
general_quirky 10
iot_coffee 10
iot_hue_lightchange 10
iot_hue_lightoff 10
lists_createoradd 10
lists_query 10
lists_remove 10
music_likeness 10
music_query 10
news_query 10
play_audiobook 10
play_game 10
play_music 10
play_podcasts 10
play_radio 10
qa_currency 10
qa_definition 10
qa_factoid 10
recommendation_events 10
recommendation_locations 10
social_post 10
takeaway_query 10
transport_query 10
transport_ticket 10
weather_query 10

Training Hyperparameters

  • batch_size: (16, 16)
  • num_epochs: (1, 1)
  • max_steps: -1
  • sampling_strategy: oversampling
  • body_learning_rate: (2e-05, 1e-05)
  • head_learning_rate: 0.01
  • loss: CosineSimilarityLoss
  • distance_metric: cosine_distance
  • margin: 0.25
  • end_to_end: False
  • use_amp: False
  • warmup_proportion: 0.1
  • seed: 42
  • eval_max_steps: -1
  • load_best_model_at_end: True

Training Results

Epoch Step Training Loss Validation Loss
0.0001 1 0.1814 -
0.0067 50 0.1542 -
0.0134 100 0.0953 -
0.0202 150 0.0991 -
0.0269 200 0.0717 -
0.0336 250 0.0653 -
0.0403 300 0.0412 -
0.0471 350 0.0534 -
0.0538 400 0.013 -
0.0605 450 0.0567 -
0.0672 500 0.0235 -
0.0739 550 0.0086 -
0.0807 600 0.0086 -
0.0874 650 0.0786 -
0.0941 700 0.0092 -
0.1008 750 0.0081 -
0.1076 800 0.0196 -
0.1143 850 0.0138 -
0.1210 900 0.0081 -
0.1277 950 0.0295 -
0.1344 1000 0.0074 -
0.1412 1050 0.0025 -
0.1479 1100 0.0036 -
0.1546 1150 0.0021 -
0.1613 1200 0.0168 -
0.1681 1250 0.0024 -
0.1748 1300 0.0039 -
0.1815 1350 0.0155 -
0.1882 1400 0.0057 -
0.1949 1450 0.0027 -
0.2017 1500 0.0018 -
0.2084 1550 0.0012 -
0.2151 1600 0.0032 -
0.2218 1650 0.0017 -
0.2286 1700 0.0012 -
0.2353 1750 0.002 -
0.2420 1800 0.0025 -
0.2487 1850 0.0014 -
0.2554 1900 0.0033 -
0.2622 1950 0.0007 -
0.2689 2000 0.0006 -
0.2756 2050 0.001 -
0.2823 2100 0.001 -
0.2891 2150 0.0007 -
0.2958 2200 0.0011 -
0.3025 2250 0.0009 -
0.3092 2300 0.0006 -
0.3159 2350 0.001 -
0.3227 2400 0.0005 -
0.3294 2450 0.0012 -
0.3361 2500 0.0005 -
0.3428 2550 0.0007 -
0.3496 2600 0.0018 -
0.3563 2650 0.0008 -
0.3630 2700 0.0009 -
0.3697 2750 0.0007 -
0.3764 2800 0.0013 -
0.3832 2850 0.0004 -
0.3899 2900 0.0005 -
0.3966 2950 0.0005 -
0.4033 3000 0.0006 -
0.4101 3050 0.0005 -
0.4168 3100 0.0004 -
0.4235 3150 0.0007 -
0.4302 3200 0.0009 -
0.4369 3250 0.0007 -
0.4437 3300 0.0007 -
0.4504 3350 0.0004 -
0.4571 3400 0.0004 -
0.4638 3450 0.0009 -
0.4706 3500 0.0006 -
0.4773 3550 0.0006 -
0.4840 3600 0.0005 -
0.4907 3650 0.0005 -
0.4974 3700 0.0003 -
0.5042 3750 0.0004 -
0.5109 3800 0.0004 -
0.5176 3850 0.0005 -
0.5243 3900 0.0007 -
0.5311 3950 0.0005 -
0.5378 4000 0.0006 -
0.5445 4050 0.0004 -
0.5512 4100 0.0006 -
0.5579 4150 0.0005 -
0.5647 4200 0.0004 -
0.5714 4250 0.0003 -
0.5781 4300 0.0003 -
0.5848 4350 0.0005 -
0.5916 4400 0.0002 -
0.5983 4450 0.0006 -
0.6050 4500 0.0004 -
0.6117 4550 0.0005 -
0.6184 4600 0.0003 -
0.6252 4650 0.0005 -
0.6319 4700 0.0007 -
0.6386 4750 0.0003 -
0.6453 4800 0.0004 -
0.6521 4850 0.0004 -
0.6588 4900 0.0004 -
0.6655 4950 0.0003 -
0.6722 5000 0.0003 -
0.6789 5050 0.0004 -
0.6857 5100 0.0003 -
0.6924 5150 0.0005 -
0.6991 5200 0.0002 -
0.7058 5250 0.0004 -
0.7126 5300 0.0003 -
0.7193 5350 0.0007 -
0.7260 5400 0.0002 -
0.7327 5450 0.0002 -
0.7394 5500 0.0005 -
0.7462 5550 0.0003 -
0.7529 5600 0.0003 -
0.7596 5650 0.0003 -
0.7663 5700 0.0004 -
0.7731 5750 0.0004 -
0.7798 5800 0.0004 -
0.7865 5850 0.0003 -
0.7932 5900 0.0003 -
0.7999 5950 0.0004 -
0.8067 6000 0.0004 -
0.8134 6050 0.0004 -
0.8201 6100 0.0003 -
0.8268 6150 0.0002 -
0.8336 6200 0.0005 -
0.8403 6250 0.0003 -
0.8470 6300 0.0003 -
0.8537 6350 0.0002 -
0.8604 6400 0.0003 -
0.8672 6450 0.0004 -
0.8739 6500 0.0002 -
0.8806 6550 0.0003 -
0.8873 6600 0.0003 -
0.8941 6650 0.0002 -
0.9008 6700 0.0002 -
0.9075 6750 0.0002 -
0.9142 6800 0.0002 -
0.9209 6850 0.0003 -
0.9277 6900 0.0002 -
0.9344 6950 0.0002 -
0.9411 7000 0.0002 -
0.9478 7050 0.0002 -
0.9546 7100 0.0002 -
0.9613 7150 0.0003 -
0.9680 7200 0.0002 -
0.9747 7250 0.0003 -
0.9814 7300 0.0002 -
0.9882 7350 0.0003 -
0.9949 7400 0.0003 -
1.0 7438 - 0.0755
  • The bold row denotes the saved checkpoint.

Framework Versions

  • Python: 3.10.13
  • SetFit: 1.0.3
  • Sentence Transformers: 2.7.0
  • Transformers: 4.39.3
  • PyTorch: 2.1.2
  • Datasets: 2.18.0
  • Tokenizers: 0.15.2

Citation

BibTeX 

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}
Downloads last month
2
Safetensors
Model size
109M params
Tensor type
F32
·
Inference Examples
Text Classification
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for aisuko/st-mpnet-v2-amazon-mi

Base model

sentence-transformers/paraphrase-mpnet-base-v2
Finetuned
(246)
this model

Evaluation results

View on Papers With Code