README.md

metadata

base_model: BAAI/bge-base-en-v1.5
library_name: setfit
metrics:
  - accuracy
pipeline_tag: text-classification
tags:
  - setfit
  - sentence-transformers
  - text-classification
  - generated_from_setfit_trainer
widget:
  - text: >-
      Reasoning for evaluation:


      **Good Points:**

      1. **Context Grounding:** The answer accurately describes many of the
      identifying characteristics of a funnel spider, such as body color, hair
      coverage, shiny carapace, and large fangs, which are all well-supported
      and mentioned in the provided document.

      2. **Relevance:** The answer directly addresses the question, which is
      about identifying a funnel spider.


      **Bad Points:**

      1. **Omissions:** The answer neglects some critical identifying details
      such as the spider's size, visible spinnerets, gender differences,
      geographical location (Australia), their hiding spots, the structure of
      their web, and some behavioral aspects, all of which were documented and
      could help in identification.

      2. **Conciseness:** Although the answer is concise, some important
      information from the document that would make the identification more
      comprehensive is missing. 


      Final Result: 

      **Bad**


      The answer, while accurate on the points it covers, is incomplete and
      misses several key identifying characteristics found in the document.
  - text: >-
      Reasoning why the answer may be good:

      1. **Context Grounding**: The answer explains specific rules and
      guidelines for writing a paper in MLA format, which seems consistent with
      several elements mentioned in the document.

      2. **Relevance**: The response directly addresses the question of how to
      write in MLA format by covering essential formatting elements such as
      margins, font size, spacing, headers, and headings.

      3. **Conciseness**: The answer is relatively concise and avoids
      overloading the reader with too much superfluous information.


      Reasoning why the answer may be bad:

      1. **Context Grounding**: The document mentions specific instructions
      about not needing a cover page and sometimes requiring one, but the
      provided answer does not acknowledge this. There is also more information
      in the document, like rules about capitalization in titles, which is
      missing.

      2. **Relevance**: Additional detail concerning specific conditions (like
      capitalizing major words in titles) could make it more comprehensive.

      3. **Conciseness**: The answer is quite thorough, but some redundant
      instructions could be streamlined further, especially related to the
      heading and title formatting.


      Final Result: 

      ****
  - text: >-
      Reasoning why the answer may be good:

      1. **Context Grounding**: The answer is supported by information derived
      from the document, specifically mentioning the importance of grades in
      core scientific subjects (Biology, Chemistry, Physics, and Mathematics)
      and the need to gain clinical experience.

      2. **Relevance**: It addresses the specific question by providing concrete
      steps on prerequisites, clinical experience, and preparation for the MCAT,
      which are relevant components of the medical school admission process.

      3. **Conciseness**: The answer is fairly clear and to the point, covering
      essential aspects without delving too deeply into extraneous details.


      Reasoning why the answer may be bad:

      1. **Context Grounding**: While the answer touches on key points, it omits
      some details from the provided document that could enhance its
      comprehensiveness, such as the importance of a well-rounded college
      experience and other preparatory steps.

      2. **Relevance**: The answer is somewhat limited in scope by not
      addressing some specific elements mentioned in the document, like the
      necessity of psychology and sociology courses for the MCAT.

      3. **Conciseness**: The answer does avoid unnecessary information but
      could be seen as overly simplistic, potentially missing the nuance and
      depthprovided in the document.


      Final Result: ****
  - text: >-
      Reasoning why the answer may be good:

      1. **Context Grounding:** The answer leverages specific strategies that
      are mentioned in the provided document, such as hiding in a laundry basket
      and picking a hiding place after the seeker checks a room.

      2. **Relevance:** The strategies suggested are directly related to
      becoming a master at hide and seek, which is exactly what the question
      asks.

      3. **Conciseness:** The answer is relatively focused and includes several
      practical tips without excessive elaboration.


      Reasoning why the answer may be bad:

      1. **Context Grounding:** It misses other valuable ideas from the document
      like using long edges, curtains, yard hiding spots, and decoys, which
      could provide a much more comprehensive answer.

      2. **Relevance:** While the answer is relevant, it introduces examples not
      as prominently detailed in the document, like hiding in plain sight behind
      multi-colored areas.

      3. **Conciseness:** Some elements could be interpreted as slightly
      redundant or not entirely necessary, such as the additional example of
      standing out behind a red couch.


      Final result: ****
  - text: >-
      Reasoning why the answer may be good:

      1. **Context Grounding**: The answer provides specific instructions on
      making and administering a saline solution to treat a baby's cough, which
      is directly mentioned in the document.

      2. **Relevance**: The answer addresses the question of how to treat a
      baby's cough by focusing on a practical method that is discussed in the
      document.

      3. **Conciseness**: The answer includes comprehensive steps to make and
      use a saline solution, which are clear and detail-oriented.


      Reasoning why the answer may be bad:

      1. **Context Grounding**: The instructions for creating the saline
      solution (2 cups of water, a tablespoon of salt, and a tablespoon of
      baking soda) are incorrect according to the document (1 cup of water, 1/2
      teaspoon of salt, and 1/2 teaspoon of baking soda).

      2. **Relevance**: The provided answer focuses only on one method (saline
      solution) and does not mention any other treatments from the document,
      making it incomplete.

      3. **Conciseness**: The answer is detailed about saline solution
      preparation and administration but includes errors, leading to an
      incorrect representation of the full range of guidelines provided in the
      document.


      Final result: ****
inference: true
model-index:
  - name: SetFit with BAAI/bge-base-en-v1.5
    results:
      - task:
          type: text-classification
          name: Text Classification
        dataset:
          name: Unknown
          type: unknown
          split: test
        metrics:
          - type: accuracy
            value: 0.7567567567567568
            name: Accuracy

SetFit with BAAI/bge-base-en-v1.5

This is a SetFit model that can be used for Text Classification. This SetFit model uses BAAI/bge-base-en-v1.5 as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

1. Fine-tuning a Sentence Transformer with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

• Model Type: SetFit
• Sentence Transformer body: BAAI/bge-base-en-v1.5
• Classification head: a LogisticRegression instance
• Maximum Sequence Length: 512 tokens
• Number of Classes: 2 classes

Model Sources

• Repository: SetFit on GitHub
• Paper: Efficient Few-Shot Learning Without Prompts
• Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
0	'Reasoning why the answer may be good:\n1. The answer covers several important aspects of petting a bearded dragon mentioned in the document, such as using slow movements, using 1 or 2 fingers to stroke the head, and using treats to encourage interaction.\n2. It also mentions key safety practices such as washing hands before and after handling the dragon.\n\nReasoning why the answer may be bad:\n1. The answer includes information ("consistently using a specific perfume or scent...") that is incorrect and not supported by the document.\n2. Some details are omitted from the answer, such as avoiding overhead movements, not petting the dragon when it’s showing signs of stress (like hissing or beard fluffing), and how to handle droopy-eyed dragons properly.\n\nFinal Result:' "Reasoning for the Evaluation:\n\nWhy the answer may be good:\n1. Context Grounding: The answer attempts to provide details on how to identify a funnel spider, which is aligned with the document.\n2. Relevance: The answer focuses on physical characteristics of funnel spiders, which relates to the identification task.\n3. Conciseness: The answer stays on topic and attempts to be straightforward without excessive information.\n\nWhy the answer may be bad:\n1. Context Grounding: The answer provides incorrect details about the funnel spider's appearance, which contradicts the provided document. For instance, it wrongly claims that the spider is light brown or gray and has non-poisonous fangs pointing sideways.\n2. Relevance: Although intended to address the question, the information presented is factually incorrect based on the provided document.\n3. Conciseness: The clarity of the answer is undermined by the inclusion of incorrect descriptions, leading to potential confusion.\n\nFinal Result: \n****" 'The given answer is:\n\n"Luis Figo left Barcelona to join Real Madrid."\n\nReasoning why the answer may be good:\n- None. The answer is completely unrelated to the question asked.\n\nReasoning why the answer may be bad:\n- Context Grounding: The answer is not grounded in the context of the provided document as it does not address the topic of real estate commissions at all.\n- Relevance: The answer does not address the specific question asked, which is about calculating real estate commissions.\n- Conciseness: Although the answer is concise, it is irrelevant and does not provide any related information to the question.\n\nFinal result:'
1	'Reasoning why the answer may be good:\n1. Context Grounding: The answer draws from multiple techniques mentioned in the document, such as quick steady breathing, good posture, engaging stomach muscles, and controlling air release.\n2. Relevance: The answer addresses the specific question of how to hold a note, detailing methods directly related to this objective.\n3. Conciseness: The information is clear, practical, and to the point without diverging into irrelevant or overly detailed explanations.\n\nReasoning why the answer may be bad:\n- Some advice in the answer (e.g., "push out your voice with your sternum") is overly simplified or not explicitly stated in the document.\n- Slight deviation into techniques like "breathe in quickly and steadily throughout the song" could be misinterpreted as contradictory to the document's suggestion of controlled breaths.\n\nFinal Result: ****' 'Reasoning Why the Answer May Be Good:\n1. Context Grounding: The answer uses multiple suggestions directly mentioned in the document, such as journaling, trying new things, and making new friends, thus grounding it in the provided context.\n2. Relevance: The response addresses the question "How to Stop Feeling Empty" by giving actionable methods to combat feelings of emptiness.\n3. Conciseness: The answer is relatively clear and to the point, focusing on specific actions one can take to alleviate empty feelings.\n\nReasoning Why the Answer May Be Bad:\n1. Context Grounding: While the answer does pull from the document, it does not cover the breadth of strategies included in the source text, potentially missing out on some helpful suggestions like adopting a pet or seeking professional help.\n2. Relevance: The answer remains relevant but does not touch on the more in-depth solutions and causes of emptiness mentioned in the document, such as mental health issues or grief, which might be critical for some readers.\n3. Conciseness: The answer is concise but perhaps overly so. It sacrifices depth for brevity, therefore not fully leveraging all the comprehensive advice provided in the document.\n\nFinal Result:' 'Reasoning why the answer may be good:\n1. Context Grounding: The provided answer mentions steps that are in the provided document, such as squeezing out excess water, applying a leave-in conditioner, and detangling with a wide-tooth comb.\n2. Relevance: The answer directly addresses the process of drying curly hair, which is the question asked.\n3. Conciseness: The answer is concise and breaks down the steps succinctly without diving into overly detailed procedures or reasons behind each step.\n\nReasoning why the answer may be bad:\n1. Context Grounding: The answer does not explicitly mention the steps found in the document about using specific conditioners, the use of T-shirts, or the avoidance of towels which were given in detail in the document.\n2. Relevance: While it mentions the general steps, it does not discuss other relevant crucial aspects such as use of anti-frizz and curling products, the method of parting and scrunching hair, and the importance of not touching hair while it dries.\n3. Conciseness: Although concise, the conciseness may bypass significant details that are crucial to comprehensive understanding, such as the reasoning behind each step and the importance ofspecific methods and products.\n\nFinal Result:'

Evaluation

Metrics

Label	Accuracy
all	0.7568

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("Netta1994/setfit_baai_wikisum_gpt-4o_improved-cot-instructions_two_reasoning_remove_final_evalu")
# Run inference
preds = model("Reasoning for evaluation:

**Good Points:**
1. **Context Grounding:** The answer accurately describes many of the identifying characteristics of a funnel spider, such as body color, hair coverage, shiny carapace, and large fangs, which are all well-supported and mentioned in the provided document.
2. **Relevance:** The answer directly addresses the question, which is about identifying a funnel spider.

**Bad Points:**
1. **Omissions:** The answer neglects some critical identifying details such as the spider's size, visible spinnerets, gender differences, geographical location (Australia), their hiding spots, the structure of their web, and some behavioral aspects, all of which were documented and could help in identification.
2. **Conciseness:** Although the answer is concise, some important information from the document that would make the identification more comprehensive is missing. 

Final Result: 
**Bad**

The answer, while accurate on the points it covers, is incomplete and misses several key identifying characteristics found in the document.")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	67	151.4225	212

Label	Training Sample Count
0	34
1	37

Training Hyperparameters

• batch_size: (16, 16)
• num_epochs: (1, 1)
• max_steps: -1
• sampling_strategy: oversampling
• num_iterations: 20
• body_learning_rate: (2e-05, 2e-05)
• head_learning_rate: 2e-05
• loss: CosineSimilarityLoss
• distance_metric: cosine_distance
• margin: 0.25
• end_to_end: False
• use_amp: False
• warmup_proportion: 0.1
• l2_weight: 0.01
• seed: 42
• eval_max_steps: -1
• load_best_model_at_end: False

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0056	1	0.2123	-
0.2809	50	0.2521	-
0.5618	100	0.1456	-
0.8427	150	0.0191	-

Framework Versions

• Python: 3.10.14
• SetFit: 1.1.0
• Sentence Transformers: 3.1.0
• Transformers: 4.44.0
• PyTorch: 2.4.1+cu121
• Datasets: 2.19.2
• Tokenizers: 0.19.1

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}