Netta1994/setfit_baai_squad_gpt-4o_cot-few_shot_remove_final_evaluation_e1_larger_train_1727019

SetFit with BAAI/bge-base-en-v1.5

This is a SetFit model that can be used for Text Classification. This SetFit model uses BAAI/bge-base-en-v1.5 as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

1. Fine-tuning a Sentence Transformer with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

• Model Type: SetFit
• Sentence Transformer body: BAAI/bge-base-en-v1.5
• Classification head: a LogisticRegression instance
• Maximum Sequence Length: 512 tokens
• Number of Classes: 2 classes

Model Sources

• Repository: SetFit on GitHub
• Paper: Efficient Few-Shot Learning Without Prompts
• Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
1	'The answer provided is clearly derived from the document and effectively summarizes Patricia Wallace’s roles and responsibilities at Oak View Elementary. It includes her tasks as the school’s social worker, coordinator of the Intervention Support Team, and attendance team leader, accurately reflecting her involvement in managing the clothing closet, food pantry, and backpack program. \n\nTherefore, the response adequately addresses the question without any extraneous or misleading information.\n\nThe final evaluation:' 'The answer is mostly accurate and covers essential steps, but it slightly misrepresents the document. It suggests making a saline solution and using it with a suction bulb, accurately detailing the saline preparation and administration. However, it could have expanded on additional tips like considering a nasal spray, maintaining humidity, and reducing dairy products, which are all present in the document.\n\nThe answer follows these steps appropriately:\n1. Make a saline solution.\n2. Administer the saline solution.\n3. Suction out the mucus.\n\nThis reflects the core advice from the document. However, the answer could be improved by mentioning more specific care instructions like keeping the air humid or how to treat any accompanying fevers.\n\nFinal evaluation:' 'The answer accurately addresses the question of identifying Toxic Shock Syndrome (TSS) and aligns with the information presented in the provided document. The key elements discussed, such as flu-like symptoms, possible associations with tampon use, the critical nature of seeking medical help, and the list of symptoms (e.g., rashes, dizziness, and disorientation), are correctlygrounded in the document.\n\nFinal evaluation:'
0	'Evaluation:\nThe provided answer incorrectly states that Fr. Zahm oversaw the creation of a literature hall, not a science hall. The document indicates that Fr. Zahm oversaw the creation of a science hall in 1883. Therefore, the answer is wrong.\n\nThe final evaluation:' "The answer accurately identifies Gregory Johnson as the CEO of Franklin Templeton Investments. It also mentions that Gregory Johnson inherited the position from his father, Rupert H. Johnson, Sr., although this latter part is not supported by the provided document. The document affirms Gregory Johnson is the current CEO but does not mention the inheritance information.\n\nThe additional detail about inheritance may be inferred from external knowledge, but strictly speaking from the provided content, the document does not validate it. However, since the question asks only for the CEO's name and the name provided is correct, the evaluation should consider the essential correctness.\n\nFinal evaluation:" 'The provided answer partially addresses the question but includes information not supported by the document. The document states that retired priests and brothers reside in Fatima House, but it does not mention that Fatima House is home to a collection of ancient religious manuscripts. Therefore, the answer introduces details not found in the document, which affects its accuracy.\n\nFinal evaluation:'

Evaluation

Metrics

Label	Accuracy
all	0.8852

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("Netta1994/setfit_baai_squad_gpt-4o_cot-few_shot_remove_final_evaluation_e1_larger_train_1727019")
# Run inference
preds = model("The answer correctly identifies that Forbes began reporting Beyoncé's annual earnings in 2008 and addresses the common misconception about Times Magazine. The information is directly supportedby the document provided.

Final evaluation:")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	16	74.1616	301

Label	Training Sample Count
0	94
1	104

Training Hyperparameters

• batch_size: (16, 16)
• num_epochs: (1, 1)
• max_steps: -1
• sampling_strategy: oversampling
• num_iterations: 20
• body_learning_rate: (2e-05, 2e-05)
• head_learning_rate: 2e-05
• loss: CosineSimilarityLoss
• distance_metric: cosine_distance
• margin: 0.25
• end_to_end: False
• use_amp: False
• warmup_proportion: 0.1
• l2_weight: 0.01
• seed: 42
• eval_max_steps: -1
• load_best_model_at_end: False

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0020	1	0.2327	-
0.1010	50	0.2358	-
0.2020	100	0.0911	-
0.3030	150	0.0324	-
0.4040	200	0.0183	-
0.5051	250	0.0226	-
0.6061	300	0.0223	-
0.7071	350	0.0098	-
0.8081	400	0.0067	-
0.9091	450	0.0057	-

Framework Versions

• Python: 3.10.14
• SetFit: 1.1.0
• Sentence Transformers: 3.1.1
• Transformers: 4.44.0
• PyTorch: 2.4.0+cu121
• Datasets: 3.0.0
• Tokenizers: 0.19.1

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}