--- library_name: setfit tags: - setfit - sentence-transformers - text-classification - generated_from_setfit_trainer metrics: - accuracy widget: - text: hi invoice wheel bearing bernard kavanagh invoice thanks cathy - text: receive message order id asin product name autoshack radiator replacement mitsubishi outlander lancer awd fwd message shipment taking long service provide solely communication buyer please aware amazon never ask provide login information verify identity service receive message service request seller central login account information report message ignore request wa email helpful response need report questionable activity copyright amazon inc affiliate right reserve amazon com terry avenue north seattle wa information help protect trust safety marketplace help arbitrate potential dispute retain message buyer seller send amazon com two year include response message amazon com us filter technology protect buyer seller possible fraud message fail filter transmit want buy confidence anytime purchase product amazon com learn safe online shopping safe buying guarantee commmgrtok - text: amazon com receive message order id asin product name autoshack front drill slot brake kit rotor silver performance ceramic pad pair driver passenger side replacement chevrolet hhr fwd message return package week ago thru receive refund yet issue service provide solely communication buyer please aware amazon never ask provide login information verify identity service receive message service request seller central login account information report message ignore request wa email helpful response need mso - text: amazon com receive message order id asin product name autoshack catalytic converter direct fit passenger side replacement infiniti nissan pathfinder armada titan ar autoshack catalytic converter exhaust pipe direct fit driver side replacement infiniti nissan pathfinder armada message hello ve contact customer regard order identify order item reason receive damage defective item detail item ha arrive wa defective customer want return item please research issue contact customer respond customer please reply e mail visit seller account following link http sellercentral amazon com gp communication manager inbox html sincerely customer service department amazon com http www amazon com service provide solely communication buyer please aware amazon never ask provide login information verify identity service receive message service request seller central login account information report message ignore request wa email helpful response need - text: create repick order sale order repick order pipeline_tag: text-classification inference: true base_model: sentence-transformers/all-mpnet-base-v2 model-index: - name: SetFit with sentence-transformers/all-mpnet-base-v2 results: - task: type: text-classification name: Text Classification dataset: name: Unknown type: unknown split: test metrics: - type: accuracy value: 0.2964972866304884 name: Accuracy --- # SetFit with sentence-transformers/all-mpnet-base-v2 This is a [SetFit](https://github.com/huggingface/setfit) model that can be used for Text Classification. This SetFit model uses [sentence-transformers/all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) as the Sentence Transformer embedding model. A [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance is used for classification. The model has been trained using an efficient few-shot learning technique that involves: 1. Fine-tuning a [Sentence Transformer](https://www.sbert.net) with contrastive learning. 2. Training a classification head with features from the fine-tuned Sentence Transformer. ## Model Details ### Model Description - **Model Type:** SetFit - **Sentence Transformer body:** [sentence-transformers/all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) - **Classification head:** a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance - **Maximum Sequence Length:** 384 tokens - **Number of Classes:** 29 classes ### Model Sources - **Repository:** [SetFit on GitHub](https://github.com/huggingface/setfit) - **Paper:** [Efficient Few-Shot Learning Without Prompts](https://arxiv.org/abs/2209.11055) - **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit) ### Model Labels | Label | Examples | |:------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | __label__25 | | | __label__27 | | | __label__16 | | | __label__17 | | | __label__30 | | | __label__18 | | | __label__29 | | | __label__19 | | | __label__26 | | | __label__24 | | | __label__31 | | | __label__0 | | | __label__22 | | | __label__23 | | | __label__33 | | | __label__28 | | | __label__13 | | | __label__4 | | | __label__7 | | | __label__8 | | | __label__21 | | | __label__3 | | | __label__15 | | | __label__20 | | | __label__2 | | | __label__35 | | | __label__5 | | | __label__34 | | | __label__11 | | ## Evaluation ### Metrics | Label | Accuracy | |:--------|:---------| | **all** | 0.2965 | ## Uses ### Direct Use for Inference First install the SetFit library: ```bash pip install setfit ``` Then you can load this model and run inference. ```python from setfit import SetFitModel # Download from the 🤗 Hub model = SetFitModel.from_pretrained("pankajkmr/setfit-paraphrase-mpnet-base-v2-sst2-model2") # Run inference preds = model("create repick order sale order repick order") ``` ## Training Details ### Training Set Metrics | Training set | Min | Median | Max | |:-------------|:----|:--------|:----| | Word count | 1 | 90.1931 | 355 | | Label | Training Sample Count | |:------------|:----------------------| | __label__0 | 15 | | __label__11 | 15 | | __label__13 | 15 | | __label__15 | 15 | | __label__16 | 15 | | __label__17 | 15 | | __label__18 | 15 | | __label__19 | 15 | | __label__2 | 15 | | __label__20 | 15 | | __label__21 | 15 | | __label__22 | 15 | | __label__23 | 15 | | __label__24 | 15 | | __label__25 | 15 | | __label__26 | 15 | | __label__27 | 15 | | __label__28 | 15 | | __label__29 | 15 | | __label__3 | 15 | | __label__30 | 15 | | __label__31 | 15 | | __label__33 | 15 | | __label__34 | 15 | | __label__35 | 15 | | __label__4 | 15 | | __label__5 | 15 | | __label__7 | 15 | | __label__8 | 15 | ### Training Hyperparameters - batch_size: (8, 8) - num_epochs: (3, 3) - max_steps: -1 - sampling_strategy: oversampling - num_iterations: 20 - body_learning_rate: (2e-05, 2e-05) - head_learning_rate: 2e-05 - loss: CosineSimilarityLoss - distance_metric: cosine_distance - margin: 0.25 - end_to_end: False - use_amp: False - warmup_proportion: 0.1 - seed: 42 - eval_max_steps: -1 - load_best_model_at_end: False ### Training Results | Epoch | Step | Training Loss | Validation Loss | |:------:|:----:|:-------------:|:---------------:| | 0.0005 | 1 | 0.1608 | - | | 0.0230 | 50 | 0.293 | - | | 0.0460 | 100 | 0.2219 | - | | 0.0690 | 150 | 0.3463 | - | | 0.0920 | 200 | 0.3157 | - | | 0.1149 | 250 | 0.2663 | - | | 0.1379 | 300 | 0.1209 | - | | 0.1609 | 350 | 0.1651 | - | | 0.1839 | 400 | 0.0464 | - | | 0.2069 | 450 | 0.1462 | - | | 0.2299 | 500 | 0.2213 | - | | 0.2529 | 550 | 0.0992 | - | | 0.2759 | 600 | 0.2794 | - | | 0.2989 | 650 | 0.0496 | - | | 0.3218 | 700 | 0.0408 | - | | 0.3448 | 750 | 0.064 | - | | 0.3678 | 800 | 0.1193 | - | | 0.3908 | 850 | 0.1822 | - | | 0.4138 | 900 | 0.1966 | - | | 0.4368 | 950 | 0.1215 | - | | 0.4598 | 1000 | 0.1847 | - | | 0.4828 | 1050 | 0.1406 | - | | 0.5057 | 1100 | 0.2141 | - | | 0.5287 | 1150 | 0.1418 | - | | 0.5517 | 1200 | 0.0398 | - | | 0.5747 | 1250 | 0.1079 | - | | 0.5977 | 1300 | 0.0704 | - | | 0.6207 | 1350 | 0.0942 | - | | 0.6437 | 1400 | 0.0751 | - | | 0.6667 | 1450 | 0.1463 | - | | 0.6897 | 1500 | 0.1015 | - | | 0.7126 | 1550 | 0.104 | - | | 0.7356 | 1600 | 0.0278 | - | | 0.7586 | 1650 | 0.0897 | - | | 0.7816 | 1700 | 0.0089 | - | | 0.8046 | 1750 | 0.228 | - | | 0.8276 | 1800 | 0.0159 | - | | 0.8506 | 1850 | 0.0039 | - | | 0.8736 | 1900 | 0.0203 | - | | 0.8966 | 1950 | 0.0768 | - | | 0.9195 | 2000 | 0.0567 | - | | 0.9425 | 2050 | 0.0952 | - | | 0.9655 | 2100 | 0.0251 | - | | 0.9885 | 2150 | 0.0425 | - | | 1.0115 | 2200 | 0.0121 | - | | 1.0345 | 2250 | 0.1579 | - | | 1.0575 | 2300 | 0.0892 | - | | 1.0805 | 2350 | 0.0142 | - | | 1.1034 | 2400 | 0.1206 | - | | 1.1264 | 2450 | 0.0257 | - | | 1.1494 | 2500 | 0.102 | - | | 1.1724 | 2550 | 0.0521 | - | | 1.1954 | 2600 | 0.0273 | - | | 1.2184 | 2650 | 0.0205 | - | | 1.2414 | 2700 | 0.0179 | - | | 1.2644 | 2750 | 0.0074 | - | | 1.2874 | 2800 | 0.007 | - | | 1.3103 | 2850 | 0.1178 | - | | 1.3333 | 2900 | 0.0051 | - | | 1.3563 | 2950 | 0.1062 | - | | 1.3793 | 3000 | 0.0214 | - | | 1.4023 | 3050 | 0.0295 | - | | 1.4253 | 3100 | 0.0967 | - | | 1.4483 | 3150 | 0.0683 | - | | 1.4713 | 3200 | 0.0019 | - | | 1.4943 | 3250 | 0.1584 | - | | 1.5172 | 3300 | 0.0719 | - | | 1.5402 | 3350 | 0.0091 | - | | 1.5632 | 3400 | 0.1362 | - | | 1.5862 | 3450 | 0.055 | - | | 1.6092 | 3500 | 0.0095 | - | | 1.6322 | 3550 | 0.194 | - | | 1.6552 | 3600 | 0.004 | - | | 1.6782 | 3650 | 0.0807 | - | | 1.7011 | 3700 | 0.0566 | - | | 1.7241 | 3750 | 0.0024 | - | | 1.7471 | 3800 | 0.0374 | - | | 1.7701 | 3850 | 0.013 | - | | 1.7931 | 3900 | 0.0662 | - | | 1.8161 | 3950 | 0.0871 | - | | 1.8391 | 4000 | 0.0112 | - | | 1.8621 | 4050 | 0.03 | - | | 1.8851 | 4100 | 0.1157 | - | | 1.9080 | 4150 | 0.1204 | - | | 1.9310 | 4200 | 0.0019 | - | | 1.9540 | 4250 | 0.0083 | - | | 1.9770 | 4300 | 0.055 | - | | 2.0 | 4350 | 0.1002 | - | | 2.0230 | 4400 | 0.0335 | - | | 2.0460 | 4450 | 0.038 | - | | 2.0690 | 4500 | 0.0134 | - | | 2.0920 | 4550 | 0.042 | - | | 2.1149 | 4600 | 0.089 | - | | 2.1379 | 4650 | 0.0408 | - | | 2.1609 | 4700 | 0.0022 | - | | 2.1839 | 4750 | 0.118 | - | | 2.2069 | 4800 | 0.0632 | - | | 2.2299 | 4850 | 0.0046 | - | | 2.2529 | 4900 | 0.0054 | - | | 2.2759 | 4950 | 0.0159 | - | | 2.2989 | 5000 | 0.0049 | - | | 2.3218 | 5050 | 0.0032 | - | | 2.3448 | 5100 | 0.0334 | - | | 2.3678 | 5150 | 0.0104 | - | | 2.3908 | 5200 | 0.0171 | - | | 2.4138 | 5250 | 0.0723 | - | | 2.4368 | 5300 | 0.101 | - | | 2.4598 | 5350 | 0.0785 | - | | 2.4828 | 5400 | 0.0686 | - | | 2.5057 | 5450 | 0.012 | - | | 2.5287 | 5500 | 0.1446 | - | | 2.5517 | 5550 | 0.032 | - | | 2.5747 | 5600 | 0.0022 | - | | 2.5977 | 5650 | 0.0127 | - | | 2.6207 | 5700 | 0.1638 | - | | 2.6437 | 5750 | 0.0039 | - | | 2.6667 | 5800 | 0.0242 | - | | 2.6897 | 5850 | 0.0337 | - | | 2.7126 | 5900 | 0.0325 | - | | 2.7356 | 5950 | 0.0024 | - | | 2.7586 | 6000 | 0.0165 | - | | 2.7816 | 6050 | 0.0015 | - | | 2.8046 | 6100 | 0.0293 | - | | 2.8276 | 6150 | 0.0008 | - | | 2.8506 | 6200 | 0.0407 | - | | 2.8736 | 6250 | 0.0032 | - | | 2.8966 | 6300 | 0.0312 | - | | 2.9195 | 6350 | 0.0143 | - | | 2.9425 | 6400 | 0.0291 | - | | 2.9655 | 6450 | 0.0017 | - | | 2.9885 | 6500 | 0.1199 | - | ### Framework Versions - Python: 3.10.12 - SetFit: 1.0.1 - Sentence Transformers: 2.2.2 - Transformers: 4.36.0 - PyTorch: 2.0.0 - Datasets: 2.16.1 - Tokenizers: 0.15.0 ## Citation ### BibTeX ```bibtex @article{https://doi.org/10.48550/arxiv.2209.11055, doi = {10.48550/ARXIV.2209.11055}, url = {https://arxiv.org/abs/2209.11055}, author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren}, keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences}, title = {Efficient Few-Shot Learning Without Prompts}, publisher = {arXiv}, year = {2022}, copyright = {Creative Commons Attribution 4.0 International} } ```