metadata

library_name: setfit
tags:
  - setfit
  - sentence-transformers
  - text-classification
  - generated_from_setfit_trainer
base_model: FacebookAI/roberta-large
metrics:
  - accuracy
widget:
  - text: Just checking in, how have you been feeling since our last chat?
  - text: I’m looking forward to learning more from you.
  - text: Take it easy!
  - text: It was great seeing you. Let's catch up again soon!
  - text: Let’s make sure you’re not carrying too much; how are you?
pipeline_tag: text-classification
inference: true
model-index:
  - name: SetFit with FacebookAI/roberta-large
    results:
      - task:
          type: text-classification
          name: Text Classification
        dataset:
          name: Unknown
          type: unknown
          split: test
        metrics:
          - type: accuracy
            value: 0.96
            name: Accuracy

SetFit with FacebookAI/roberta-large

This is a SetFit model that can be used for Text Classification. This SetFit model uses FacebookAI/roberta-large as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

Fine-tuning a Sentence Transformer with contrastive learning.
Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

Model Type: SetFit
Sentence Transformer body: FacebookAI/roberta-large
Classification head: a LogisticRegression instance
Maximum Sequence Length: 512 tokens
Number of Classes: 2 classes

Model Sources

Repository: SetFit on GitHub
Paper: Efficient Few-Shot Learning Without Prompts
Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
true	'See you soon!' 'You look well!' 'Your journey is quite inspiring, can you share more about it?'
false	'What are the core components of your business model?' 'How do you balance your personal and professional life?' "There is a situation where a daughter of a narcissistic mother denigrated the father. When the mother complained to the daughter about the father and how poor he was a a husband and person and how badly he treated the wife. The mother's claims were inaccurate and overblown. The mother said I inappropriate things to the daughter such as he flirted with other women, or the mother could have done much better than marrying him. After such episodes, the daughter was dismissive and rude to the father. What are the signs of parental alienation and what are the impacts on a daughter growing up and as an adult?"

Label

Examples

true

'See you soon!'
'You look well!'
'Your journey is quite inspiring, can you share more about it?'

false

'What are the core components of your business model?'
'How do you balance your personal and professional life?'
"There is a situation where a daughter of a narcissistic mother denigrated the father. When the mother complained to the daughter about the father and how poor he was a a husband and person and how badly he treated the wife. The mother's claims were inaccurate and overblown. The mother said I inappropriate things to the daughter such as he flirted with other women, or the mother could have done much better than marrying him. After such episodes, the daughter was dismissive and rude to the father. What are the signs of parental alienation and what are the impacts on a daughter growing up and as an adult?"

Evaluation

Metrics

Label	Accuracy
all	0.96

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("richie-ghost/setfit-FacebookAI-roberta-large-phatic")
# Run inference
preds = model("Take it easy!")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	1	9.8722	108

Label	Training Sample Count
false	191
true	169

Training Hyperparameters

batch_size: (16, 16)
num_epochs: (4, 4)
max_steps: -1
sampling_strategy: oversampling
body_learning_rate: (2e-05, 1e-05)
head_learning_rate: 0.01
loss: CosineSimilarityLoss
distance_metric: cosine_distance
margin: 0.25
end_to_end: False
use_amp: False
warmup_proportion: 0.1
seed: 42
eval_max_steps: -1
load_best_model_at_end: True

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0002	1	0.4745	-
0.0122	50	0.441	-
0.0245	100	0.4422	-
0.0367	150	0.2339	-
0.0489	200	0.1182	-
0.0612	250	0.0806	-
0.0734	300	0.1183	-
0.0856	350	0.0551	-
0.0978	400	0.0146	-
0.1101	450	0.0115	-
0.1223	500	0.0042	-
0.1345	550	0.0053	-
0.1468	600	0.0021	-
0.1590	650	0.0596	-
0.1712	700	0.0029	-
0.1835	750	0.0009	-
0.1957	800	0.0002	-
0.2079	850	0.0005	-
0.2202	900	0.0013	-
0.2324	950	0.0008	-
0.2446	1000	0.0004	-
0.2568	1050	0.0004	-
0.2691	1100	0.0004	-
0.2813	1150	0.0003	-
0.2935	1200	0.0003	-
0.3058	1250	0.0012	-
0.3180	1300	0.0001	-
0.3302	1350	0.0002	-
0.3425	1400	0.0003	-
0.3547	1450	0.0024	-
0.3669	1500	0.0008	-
0.3792	1550	0.0015	-
0.3914	1600	0.0002	-
0.4036	1650	0.0002	-
0.4159	1700	0.1842	-
0.4281	1750	0.0009	-
0.4403	1800	0.0001	-
0.4525	1850	0.0013	-
0.4648	1900	0.0637	-
0.4770	1950	0.0002	-
0.4892	2000	0.0007	-
0.5015	2050	0.0001	-
0.5137	2100	0.0	-
0.5259	2150	0.0	-
0.5382	2200	0.0	-
0.5504	2250	0.0	-
0.5626	2300	0.0001	-
0.5749	2350	0.0	-
0.5871	2400	0.0	-
0.5993	2450	0.0	-
0.6115	2500	0.0	-
0.6238	2550	0.0	-
0.6360	2600	0.0	-
0.6482	2650	0.0	-
0.6605	2700	0.0001	-
0.6727	2750	0.0	-
0.6849	2800	0.0	-
0.6972	2850	0.0	-
0.7094	2900	0.0	-
0.7216	2950	0.0	-
0.7339	3000	0.0	-
0.7461	3050	0.0	-
0.7583	3100	0.0001	-
0.7705	3150	0.0	-
0.7828	3200	0.0	-
0.7950	3250	0.0	-
0.8072	3300	0.0	-
0.8195	3350	0.0	-
0.8317	3400	0.0	-
0.8439	3450	0.0001	-
0.8562	3500	0.0	-
0.8684	3550	0.0	-
0.8806	3600	0.0	-
0.8929	3650	0.0	-
0.9051	3700	0.0	-
0.9173	3750	0.0	-
0.9295	3800	0.0	-
0.9418	3850	0.0	-
0.9540	3900	0.0	-
0.9662	3950	0.0	-
0.9785	4000	0.0	-
0.9907	4050	0.0	-
1.0	4088	-	0.0815
1.0029	4100	0.0	-
1.0152	4150	0.0	-
1.0274	4200	0.0	-
1.0396	4250	0.0	-
1.0519	4300	0.0	-
1.0641	4350	0.0	-
1.0763	4400	0.0	-
1.0886	4450	0.0	-
1.1008	4500	0.0	-
1.1130	4550	0.0	-
1.1252	4600	0.0	-
1.1375	4650	0.0	-
1.1497	4700	0.0	-
1.1619	4750	0.0	-
1.1742	4800	0.0	-
1.1864	4850	0.0	-
1.1986	4900	0.0	-
1.2109	4950	0.0	-
1.2231	5000	0.0	-
1.2353	5050	0.0	-
1.2476	5100	0.0	-
1.2598	5150	0.0	-
1.2720	5200	0.0	-
1.2842	5250	0.0	-
1.2965	5300	0.0	-
1.3087	5350	0.0	-
1.3209	5400	0.0	-
1.3332	5450	0.0	-
1.3454	5500	0.0	-
1.3576	5550	0.0	-
1.3699	5600	0.0	-
1.3821	5650	0.0	-
1.3943	5700	0.0	-
1.4066	5750	0.0	-
1.4188	5800	0.0	-
1.4310	5850	0.0	-
1.4432	5900	0.0	-
1.4555	5950	0.0	-
1.4677	6000	0.0	-
1.4799	6050	0.0	-
1.4922	6100	0.0	-
1.5044	6150	0.0112	-
1.5166	6200	0.4712	-
1.5289	6250	0.3977	-
1.5411	6300	0.2112	-
1.5533	6350	0.318	-
1.5656	6400	0.2523	-
1.5778	6450	0.2829	-
1.5900	6500	0.2736	-
1.6023	6550	0.2493	-
1.6145	6600	0.3112	-
1.6267	6650	0.2291	-
1.6389	6700	0.2855	-
1.6512	6750	0.2642	-
1.6634	6800	0.2376	-
1.6756	6850	0.2983	-
1.6879	6900	0.2853	-
1.7001	6950	0.3095	-
1.7123	7000	0.2497	-
1.7246	7050	0.2305	-
1.7368	7100	0.2433	-
1.7490	7150	0.2505	-
1.7613	7200	0.2292	-
1.7735	7250	0.3028	-
1.7857	7300	0.2394	-
1.7979	7350	0.2601	-
1.8102	7400	0.2417	-
1.8224	7450	0.2086	-
1.8346	7500	0.2573	-
1.8469	7550	0.2344	-
1.8591	7600	0.2381	-
1.8713	7650	0.2772	-
1.8836	7700	0.2614	-
1.8958	7750	0.2659	-
1.9080	7800	0.2536	-
1.9203	7850	0.2385	-
1.9325	7900	0.2695	-
1.9447	7950	0.2512	-
1.9569	8000	0.2216	-
1.9692	8050	0.2291	-
1.9814	8100	0.2443	-
1.9936	8150	0.2579	-
2.0	8176	-	0.5
2.0059	8200	0.2605	-
2.0181	8250	0.2528	-
2.0303	8300	0.2361	-
2.0426	8350	0.2891	-
2.0548	8400	0.2692	-
2.0670	8450	0.25	-
2.0793	8500	0.2362	-
2.0915	8550	0.2833	-
2.1037	8600	0.2698	-
2.1159	8650	0.2195	-
2.1282	8700	0.2621	-
2.1404	8750	0.2564	-
2.1526	8800	0.2657	-
2.1649	8850	0.2629	-
2.1771	8900	0.2503	-
2.1893	8950	0.2583	-
2.2016	9000	0.2694	-
2.2138	9050	0.2824	-
2.2260	9100	0.2675	-
2.2383	9150	0.2699	-
2.2505	9200	0.2515	-
2.2627	9250	0.2511	-
2.2750	9300	0.2518	-
2.2872	9350	0.2555	-
2.2994	9400	0.2512	-
2.3116	9450	0.2374	-
2.3239	9500	0.2546	-
2.3361	9550	0.2846	-
2.3483	9600	0.2617	-
2.3606	9650	0.2474	-
2.3728	9700	0.2454	-
2.3850	9750	0.2265	-
2.3973	9800	0.2272	-
2.4095	9850	0.2442	-
2.4217	9900	0.236	-
2.4340	9950	0.2382	-
2.4462	10000	0.2645	-
2.4584	10050	0.2707	-
2.4706	10100	0.2573	-
2.4829	10150	0.2435	-
2.4951	10200	0.2705	-
2.5073	10250	0.2808	-
2.5196	10300	0.2581	-
2.5318	10350	0.2544	-
2.5440	10400	0.2333	-
2.5563	10450	0.2544	-
2.5685	10500	0.2497	-
2.5807	10550	0.2575	-
2.5930	10600	0.2382	-
2.6052	10650	0.2451	-
2.6174	10700	0.2702	-
2.6296	10750	0.2569	-
2.6419	10800	0.249	-
2.6541	10850	0.2366	-
2.6663	10900	0.2278	-
2.6786	10950	0.2568	-
2.6908	11000	0.2721	-
2.7030	11050	0.2593	-
2.7153	11100	0.2439	-
2.7275	11150	0.2543	-
2.7397	11200	0.2478	-
2.7520	11250	0.2325	-
2.7642	11300	0.2538	-
2.7764	11350	0.2968	-
2.7886	11400	0.2505	-
2.8009	11450	0.2377	-
2.8131	11500	0.2547	-
2.8253	11550	0.2529	-
2.8376	11600	0.2502	-
2.8498	11650	0.2293	-
2.8620	11700	0.2676	-
2.8743	11750	0.2371	-
2.8865	11800	0.2495	-
2.8987	11850	0.2937	-
2.9110	11900	0.2355	-
2.9232	11950	0.2482	-
2.9354	12000	0.2336	-
2.9477	12050	0.2344	-
2.9599	12100	0.257	-
2.9721	12150	0.2557	-
2.9843	12200	0.2854	-
2.9966	12250	0.2455	-
3.0	12264	-	0.5
3.0088	12300	0.2323	-
3.0210	12350	0.2566	-
3.0333	12400	0.2319	-
3.0455	12450	0.2552	-
3.0577	12500	0.2796	-
3.0700	12550	0.2823	-
3.0822	12600	0.2303	-
3.0944	12650	0.2448	-
3.1067	12700	0.2502	-
3.1189	12750	0.2516	-
3.1311	12800	0.2537	-
3.1433	12850	0.251	-
3.1556	12900	0.2639	-
3.1678	12950	0.2321	-
3.1800	13000	0.282	-
3.1923	13050	0.2577	-
3.2045	13100	0.2448	-
3.2167	13150	0.2352	-
3.2290	13200	0.281	-
3.2412	13250	0.2337	-
3.2534	13300	0.268	-
3.2657	13350	0.261	-
3.2779	13400	0.2378	-
3.2901	13450	0.2588	-
3.3023	13500	0.266	-
3.3146	13550	0.2604	-
3.3268	13600	0.2202	-
3.3390	13650	0.2217	-
3.3513	13700	0.2464	-
3.3635	13750	0.2684	-
3.3757	13800	0.2279	-
3.3880	13850	0.2379	-
3.4002	13900	0.2741	-
3.4124	13950	0.2713	-
3.4247	14000	0.2581	-
3.4369	14050	0.2638	-
3.4491	14100	0.2125	-
3.4614	14150	0.2348	-
3.4736	14200	0.2253	-
3.4858	14250	0.2627	-
3.4980	14300	0.2463	-
3.5103	14350	0.2533	-
3.5225	14400	0.2422	-
3.5347	14450	0.2296	-
3.5470	14500	0.2532	-
3.5592	14550	0.2733	-
3.5714	14600	0.2258	-
3.5837	14650	0.2253	-
3.5959	14700	0.2388	-
3.6081	14750	0.2217	-
3.6204	14800	0.3033	-
3.6326	14850	0.2349	-
3.6448	14900	0.2596	-
3.6570	14950	0.2415	-
3.6693	15000	0.2494	-
3.6815	15050	0.2826	-
3.6937	15100	0.2633	-
3.7060	15150	0.2636	-
3.7182	15200	0.2351	-
3.7304	15250	0.264	-
3.7427	15300	0.2652	-
3.7549	15350	0.2724	-
3.7671	15400	0.2731	-
3.7794	15450	0.2825	-
3.7916	15500	0.2611	-
3.8038	15550	0.2574	-
3.8160	15600	0.261	-
3.8283	15650	0.219	-
3.8405	15700	0.2323	-
3.8527	15750	0.2442	-
3.8650	15800	0.2509	-
3.8772	15850	0.26	-
3.8894	15900	0.2475	-
3.9017	15950	0.2452	-
3.9139	16000	0.2598	-
3.9261	16050	0.2377	-
3.9384	16100	0.2445	-
3.9506	16150	0.2451	-
3.9628	16200	0.2714	-
3.9750	16250	0.2755	-
3.9873	16300	0.2579	-
3.9995	16350	0.2338	-
4.0	16352	-	0.5

The bold row denotes the saved checkpoint.

Framework Versions

Python: 3.10.12
SetFit: 1.0.3
Sentence Transformers: 2.7.0
Transformers: 4.40.0
PyTorch: 2.2.1+cu121
Datasets: 2.19.0
Tokenizers: 0.19.1

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}