Add SetFit model

README.md

@@ -87,7 +87,7 @@ model-index:
       split: test
     metrics:
     - type: accuracy
-      value: 0.84
+      value: 0.88
       name: Accuracy
 ---
 
@@ -119,17 +119,17 @@ The model has been trained using an efficient few-shot learning technique that i
 - **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit)
 
 ### Model Labels
-| Label | Examples                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
-|:------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| 0     | <ul><li>"Reasoning:\nThe answer provided closely aligns with the specific instructions given in the document about petting a bearded dragon. It correctly mentions using 1 or 2 fingers to gently stroke the dragon's head, lowering your hand slowly to avoid startling it, and washing hands before and after petting to reduce the risk of bacteria transfer. However, the part about using a specific perfume or scent to help the dragon recognize you is not supported by the text and is, in fact, incorrect.\n\nFinal Evaluation: \nResult:"</li><li>'Reasoning:\nThe answer provided addresses the physical characteristics of a funnel spider but includes several inaccuracies and deviations from the information in the provided document. Key errors include describing the funnel spider as light brown or gray with a soft, dull carapace, which contradicts the document’s description of a dark brown or black body and a hard, shiny carapace. Additionally, the claim that funnel spiders have 3 non-poisonous fangs pointing sideways is incorrect based on the document, which states that the funnel spider has two large, downward-pointing fangs that are poisonous. The document provides clear and detailed descriptions that should form the basis for an accurate answer.\n\nFinal Evaluation:'</li><li>'Reasoning:\nThe answer provided, "Luis Figo left Barcelona to join Real Madrid," while factually correct according to the provided document, is entirely unrelated to the question "How to Calculate Real Estate Commissions." The document and the answer focus on a historical event in soccer rather than providing any information or calculations related to real estate commissions. \n\nFinal Evaluation:'</li></ul> |
-| 1     | <ul><li>'Reasoning:\nThe answer is well-supported by the document and directly relates to the question of how to hold a note while singing. It addresses key aspects such as breathing techniques, posture, and controlled release of air, all of which are mentioned in the provided document. The answer stays concise and clear, without deviating into unrelated topics, effectively summarizing the necessary steps for holding a note.\n\nFinal result:'</li><li>'Reasoning:\nThe answer is well-founded in the provided document and directly relates to the question of how to stop feeling empty. It suggests practical actions like keeping a journal, trying new activities, and making new friends, all of which are discussed in the document. The recommendations in the answer are summarized clearly and are appropriate responses to the question without providing extraneous information.\n\nFinal Evaluation:'</li><li>'Reasoning:\nThe answer aligns well with the instructions provided in the document and effectively addresses the question of how to dry curly hair. It begins by recommending gently squeezing out excess water, followed by the application of a leave-in conditioner and the use of a wide-tooth comb for detangling, which are all steps mentioned in the document. The answer then advises adding styling products and parting the hair to lift the roots, which helps expedite the air-drying process. The key points from the document are reflected in the answer, ensuring it is contextually grounded and relevant.\n\nEvaluation:'</li></ul>                                                                                                                                                                 |
+| Label | Examples                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
+|:------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| 1     | <ul><li>'Reasoning:\nThe answer correctly identifies Joan Gaspart as the individual who resigned from the presidency of Barcelona after the team\'s poor showing in the 2003 season. This is directly supported by the document, which explicitly states that "club president Joan Gaspart resigned, his position having been made completely untenable by such a disastrous season on top of the club\'s overall decline in fortunes since he became president three years prior." The answer is concise and directly relevant to the question without including any extraneous information.\n\nEvaluation:'</li><li>"Reasoning:\nThe provided answer directly addresses the question of why it is recommended to hire a professional residential electrician like O'Hara Electric for electrical work in your house. The answer highlights key points such as the hazards of working with electricity, the potential for injury, and the long-term implications of improperly done electrical work. It also mentions the risk involved even in seemingly simple tasks like smoke detector installation and emphasizes the benefits of having the job done correctly the first time by a professional. The details arewell-supported by the document.\n\nEvaluation:"</li><li>'Reasoning:\nThe answer "The title of Aerosmith\'s 1987 comeback album was \'Permanent Vacation\'" is directly supported by the provided document. The document explicitly states, "Aerosmith\'s comeback album Permanent Vacation (1987) would begin a decade long revival of their popularity." The answer is directly related to the question asked and does not deviate into unrelated topics, ensuring conciseness and relevance.\n\nEvaluation:'</li></ul>                                                       |
+| 0     | <ul><li>'Reasoning:\nThe answer provides a well-supported response that aligns directly with the content presented in the document. It addresses various strategies to combat smoking cravings, such as identifying and avoiding triggers, using distractions, and engaging in alternative activities. Specific triggers, like daily routines and social situations, are described in both the answer and the document. Additionally, the advice on using chewing licorice root and engaging in smoke-free activities is related to the suggestions given in the document. The answer is clear, concise, and stays relevant to the question throughout.\n\nFinal Evaluation: \nEvaluation:'</li><li>"Reasoning:\nThe provided answer accurately captures the challenges Amy Bloom faces when starting a significant writing project, as detailed in the document. Notably, it mentions the difficulty of getting started, the need to clear mental space, and to recalibrate her daily life, which are all points grounded in the text. The answer also covers her becoming less involved in everyday life and spending less time on domestic concerns, which aligns well with the provided passage. However, the part about traveling to a remote island with no internet access is not mentioned in the document and appears to be fabricated, which detracts from the answer's context grounding.\n\nFinal Result:"</li><li>'Reasoning:\nThe provided answer incorrectly states the price and location of the 6 bedroom detached house. According to the document, the 6 bedroom detached house is for sale at a price of £950,000 and is located at Willow Drive, Twyford, Reading, Berkshire, RG10. The answer gives a different priceand an incorrect location.\n\nFinal Evaluation:'</li></ul> |
 
 ## Evaluation
 
 ### Metrics
 | Label   | Accuracy |
 |:--------|:---------|
-| **all** | 0.84     |
+| **all** | 0.88     |
 
 ## Uses
 
@@ -183,12 +183,12 @@ Evaluation:")
 ### Training Set Metrics
 | Training set | Min | Median  | Max |
 |:-------------|:----|:--------|:----|
-| Word count   | 57  | 92.5070 | 176 |
+| Word count   | 33  | 76.9045 | 176 |
 
 | Label | Training Sample Count |
 |:------|:----------------------|
-| 0     | 34                    |
-| 1     | 37                    |
+| 0     | 95                    |
+| 1     | 104                   |
 
 ### Training Hyperparameters
 - batch_size: (16, 16)
@@ -212,18 +212,24 @@ Evaluation:")
 ### Training Results
 | Epoch  | Step | Training Loss | Validation Loss |
 |:------:|:----:|:-------------:|:---------------:|
-| 0.0056 | 1    | 0.2159        | -               |
-| 0.2809 | 50   | 0.2444        | -               |
-| 0.5618 | 100  | 0.0815        | -               |
-| 0.8427 | 150  | 0.0041        | -               |
+| 0.0020 | 1    | 0.2375        | -               |
+| 0.1004 | 50   | 0.2548        | -               |
+| 0.2008 | 100  | 0.2339        | -               |
+| 0.3012 | 150  | 0.0973        | -               |
+| 0.4016 | 200  | 0.0347        | -               |
+| 0.5020 | 250  | 0.0125        | -               |
+| 0.6024 | 300  | 0.0058        | -               |
+| 0.7028 | 350  | 0.004         | -               |
+| 0.8032 | 400  | 0.0033        | -               |
+| 0.9036 | 450  | 0.0023        | -               |
 
 ### Framework Versions
 - Python: 3.10.14
 - SetFit: 1.1.0
-- Sentence Transformers: 3.1.0
+- Sentence Transformers: 3.1.1
 - Transformers: 4.44.0
-- PyTorch: 2.4.1+cu121
-- Datasets: 2.19.2
+- PyTorch: 2.4.0+cu121
+- Datasets: 3.0.0
 - Tokenizers: 0.19.1
 
 ## Citation

config_sentence_transformers.json

@@ -1,8 +1,8 @@
 {
   "__version__": {
-    "sentence_transformers": "3.1.0",
+    "sentence_transformers": "3.1.1",
     "transformers": "4.44.0",
-    "pytorch": "2.4.1+cu121"
+    "pytorch": "2.4.0+cu121"
   },
   "prompts": {},
   "default_prompt_name": null,

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a18f1d2fb54ceed55f839c474d0efd34b14b8a78fd1e8e16ac9d1b6f5aec7249
+oid sha256:6dba08c7ab2d819ea194d586aa90d391d26c1988c9b3964cd4eb0688657724f4
 size 437951328

model_head.pkl

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:39305b88397d03e24852c28ef118211d100db284b49c17876de857ac3b0edec4
+oid sha256:c8f3639129184be415f0a8841f273bfd5957b94e9b4973a71a8122cee2ce8718
 size 7007