metadata

base_model: BAAI/bge-base-en-v1.5
library_name: setfit
metrics:
  - accuracy
pipeline_tag: text-classification
tags:
  - setfit
  - sentence-transformers
  - text-classification
  - generated_from_setfit_trainer
widget:
  - text: >-
      Reasoning:


      1. **Context Grounding**: The provided answer does not directly reference
      where to access specific training resources. It rather gives a mixture of
      unrelated information such as using a password manager, secure file
      sharing, personal development budget, etc. These points, while valid, do
      not directly pertain to accessing training resources specifically.
         
      2. **Relevance**: The response deviates from speaking directly about
      accessing training resources and instead covers a broader range of topics,
      which includes using company systems for other purposes, security
      protocols, and personal development budget discussions. These areas do not
      directly answer the question.


      3. **Conciseness**: The answer lacks conciseness as it includes quite a
      bit of unrelated information, which makes it unnecessarily lengthy and
      potentially confusing.


      4. **Specificity**: The answer fails to be specific about how to access
      training resources. If the document does contain the relevant information,
      it hasn't been effectively pulled out or emphasized in the answer.


      5. **No-Response Principle**: If a specific question cannot be answered
      based on the given document, it should be clearly stated. Here, it seems
      the document may not contain a straightforward guide to accessing training
      resources, but this isn't clarified.


      Final Result: **Bad**
  - text: >-
      **Reasoning:**


      1. **Context Grounding:** The answer is well-supported by Document 1,
      which specifies that questions about travel reimbursement should be
      directed to finance@ORGANIZATION_2.

      2. **Relevance:** The answer is directly relevant to the question, which
      asks whom to contact about travel reimbursement.

      3. **Conciseness:** The answer is concise and directly addresses the
      question without including any unnecessary information.

      4. **Response Appropriateness:** The answer correctly utilizes the detail
      provided in the document and does not attempt to provide unnecessary
      information.

      5. **Specificity:** The answer provides the specific email address for the
      relevant point of contact.

      6. **Balanced Detail:** The answer includes the required contact
      information and is not too general.


      **Final Result: Good.**
  - text: >-
      The answer provided fulfills the criteria as outlined:


      1. **Context Grounding**: The answer accurately reflects the contents of
      the documents, particularly Document 1, which specifies the steps team
      leads should take when they consider the possibility that someone's time
      at the ORGANIZATION is up.


      2. **Relevance**: The answer directly addresses the question by explaining
      why it is important for team leads to think about the possibility of
      someone leaving, including the potential benefits of addressing issues
      before they become unmanageable.


      3. **Conciseness**: The answer is somewhat verbose but manages to stay
      mostly on-topic and addresses the question throughout, ensuring it doesn't
      veer into unrelated topics.


      4. **Specificity**: While a bit generalized in parts, the answer includes
      specifics such as underperformance, lack of growth, and disagreement with
      company direction, which are supported by the details in the documents.


      Therefore, despite minor verbosity, the answer is accurate, relevant, and
      detailed enough to be considered good.


      **Final Result: Good**
  - text: >-
      Reasoning:

      1. Context Grounding: The response draws from the documents providing
      relevant sources such as the organization's website, job ads, and
      newsletter link.

      2. Relevance: The answer is directly related to the question about
      understanding the organization's products, challenges, and future.

      3. Conciseness: The answer is clear and to the point.

      4. Does not attempt to respond when the document lacks information: It
      addresses the question appropriately with the available information.

      5. Specificity: The answer is specific and provides concrete steps to
      follow.

      6. Relevant tips: The answer includes actionable steps like visiting the
      website, viewing job ads, and signing up for a newsletter, which are
      relevant.


      The answer precisely matches all the criteria set for evaluation.


      Final Result: Good
  - text: >-
      Reasoning:

      1. Context Grounding: The answer is well-grounded in the document,
      explaining the roles of ORGANIZATION_2, Thomas Barnes, and Charlotte
      Herrera correctly based on the provided information.

      2. Relevance: The answer directly addresses the question, detailing the
      extent of ORGANIZATION_2's participation in the farewell process.

      3. Conciseness: While mostly concise, the answer could be slightly more
      succinct. Repetition of certain elements could be minimized.

      4. Appropriateness of Response: The answer does not attempt to respond
      beyond the document.

      5. Specificity: The answer is specific, describing the involvement of
      specific people and their roles.

      6. Tips and Generality: The answer includes relevant insights into the
      roles, though it does not go into overly general territory.


      Final Result: Good
inference: true
model-index:
  - name: SetFit with BAAI/bge-base-en-v1.5
    results:
      - task:
          type: text-classification
          name: Text Classification
        dataset:
          name: Unknown
          type: unknown
          split: test
        metrics:
          - type: accuracy
            value: 0.6716417910447762
            name: Accuracy

SetFit with BAAI/bge-base-en-v1.5

This is a SetFit model that can be used for Text Classification. This SetFit model uses BAAI/bge-base-en-v1.5 as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

Fine-tuning a Sentence Transformer with contrastive learning.
Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

Model Type: SetFit
Sentence Transformer body: BAAI/bge-base-en-v1.5
Classification head: a LogisticRegression instance
Maximum Sequence Length: 512 tokens
Number of Classes: 2 classes

Model Sources

Repository: SetFit on GitHub
Paper: Efficient Few-Shot Learning Without Prompts
Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
0	"Reasoning: \n\n1. Context Grounding: The answer pulls loosely from the documents provided but does not ground the response specifically to any of the documents' key points about budgeting or investment decisions.\n2. Relevance: While the query is about saving money in general, the answer ambiguously addresses saving money by suggesting wise spending and seeking guidance, which are indirectly related to what's mentioned in the documents.\n3. Conciseness: The answer contains unnecessary details like mentioning health and priorities, which are somewhat related based on Document 3 but are not directly in line with the exact scope of the question.\n4. Too General: The answer is broad and generic. The document contains specific guidelines (such as asking if the investment would be made with personal money and consulting with ORGANIZATION_2 for high-stakes cases), which the response fails to integrate effectively.\n5. Detail Inclusion: The answer needed to incorporate specific strategies mentioned in the documents, like ensuring if an expense aligns with personal and organizational benefits or seeking guidance on higher-budget items.\n\nFinal result: Bad" 'Reasoning:\n\n1. Context Grounding: The answer does mention several pet peeves directly from the provided documents, though it incorrectly repeats the names and has some formatting errors, it aligns generally with the content.\n2. Relevance: The answer focuses on the specific question about pet peeves, summarizing relevant points from the documents.\n3. Conciseness: The answer is cluttered with repeated names and formatting errors that make it unnecessarily lengthy and harder to read.\n4. Avoiding unfounded responses: The answer is founded on the document but suffers from poor editing issues.\n5. Specificity: The answer includes specific pet peeves, but it also includes unnecessary repetition.\n6. Tip Inclusion: The query did not ask for tips, so this criterion is not strictly applicable.\n\nFinal result: Bad\n\nThe repeated names and formatting errors indicate lack of editing and care in the response, making it difficult to read and comprehend fully. It does not meet the required standard for a good, concise, and clear response.' "Reasoning:\n\n1. Context Grounding: The answer only indirectly relates to the question. The provided information gives various ways to interact with the company's systems or tools rather than specifically how to access the training resources.\n2. Relevance: The answer deviates from the main question. It mentions systems for personal documents, expense claims, password management, secure sharing, feedback processes, and learning budgets, none of which directly answer how to access training resources.\n3. Conciseness: While the answer is detailed, it includes unnecessary information that isn't directly related to accessing training resources, making it not concise.\n4. Specificity: The answer lacks details on accessing training materials or platforms where these resources are housed.\n5. Generalization: It provides general guidance on several aspects but fails to narrow down specifically on training resources or offer relevant tips directly tied to them.\n\nFinal result: Bad"
1	'Reasoning:\n\n1. Context Grounding: The answer references various elements from the documents provided, such as giving feedback at the time of the event, focusing on the situation and not the person, and showing appreciation. It generally aligns with the tips described in the documents.\n \n2. Relevance: The answer directly addresses the question of how feedback should be given, touching on many of the specific points mentioned in the documents.\n\n3. Conciseness: The response is overly wordy and includes some repetition (e.g., focusing on the situation not the person and the intention behind giving feedback). It can be more concise while still covering the necessary points.\n\n4. Document Dependency: The response accurately reflects the content of the provided documents. However, it brings in names like "Christopher Estes," "emichelle James Johnson MD," which may be elements of the document's placeholders rather than relevant names in the context of feedback tips. This adds unnecessary confusion to the answer.\n\n5. Specificity: The answer appropriately includes specific advice, though it overdoes some parts (like the role and intentions in giving feedback).\n\n6. Accuracy and Clarity: There are minor grammatical errors and some parts like "emichelle James Johnson MDamples" which are not clear and probably are typos or formatting errors.\n\nGiven the evaluation above:\n\nFinal Result: Bad' 'Reasoning:\n1. Context Grounding: The answer refers to sharing information from high-level meetings which is supported by Document 4. However, the inclusion of "Robin Nichols" appears to be an error or typo, which diminishes the context grounding slightly.\n2. Relevance: The answer is relevant to the question asked; it covers the need for transparency, team alignment, addressing concerns, and fostering an open environment, which is in line with the information in Document 4.\n3. Conciseness: The answer is moderately concise but slightly verbose. It can be trimmed down while still maintaining the key points.\n4. Directness: The answer attempts to respond to the question based on the content provided in Document 4.\n5. Specificity: The answer includes specific points about decisions made and the rationale being shared, aligning the team and fostering a sense of purpose.\n6. Tips: The answer implicitly includes tips such as ensuring transparency and addressing team concerns.\n\nFinal Result: Good' 'Reasoning:\n\n1. Context Grounding: The given answer closely follows the instructions provided in Document 1 regarding car travel expenses. The essential details, such as keeping track of kilometers and submitting an Excel document or email to specified email addresses, are accurately mentioned.\n \n2. Relevance: The answer is pertinent to the question and provides specific steps on how to report car travel expenses, which is exactly what was asked.\n\n3. Conciseness: The response is generally concise, though it includes a minor unnecessary detail about requesting a parking card for a specific date which doesn’t directly pertain to the monthly reporting of car travel expenses.\n\n4. Specificity: The answer effectively covers the key points mentioned in the document for reporting travel expenses, including the specific reimbursement rate.\n\n5. General Tips: The tip about requesting a parking card is accurate but not directly relevant to the primary question of how to report car travel expenses.\n\nGiven the accurate detailing and mostly relevant content, while removing the unnecessary detail about the parking card date would improve the response, the answer is still sufficiently useful.\n\nFinal Result: Good'

Label

Examples

"Reasoning: \n\n1. Context Grounding: The answer pulls loosely from the documents provided but does not ground the response specifically to any of the documents' key points about budgeting or investment decisions.\n2. Relevance: While the query is about saving money in general, the answer ambiguously addresses saving money by suggesting wise spending and seeking guidance, which are indirectly related to what's mentioned in the documents.\n3. Conciseness: The answer contains unnecessary details like mentioning health and priorities, which are somewhat related based on Document 3 but are not directly in line with the exact scope of the question.\n4. Too General: The answer is broad and generic. The document contains specific guidelines (such as asking if the investment would be made with personal money and consulting with ORGANIZATION_2 for high-stakes cases), which the response fails to integrate effectively.\n5. Detail Inclusion: The answer needed to incorporate specific strategies mentioned in the documents, like ensuring if an expense aligns with personal and organizational benefits or seeking guidance on higher-budget items.\n\nFinal result: Bad"
'Reasoning:\n\n1. Context Grounding: The answer does mention several pet peeves directly from the provided documents, though it incorrectly repeats the names and has some formatting errors, it aligns generally with the content.\n2. Relevance: The answer focuses on the specific question about pet peeves, summarizing relevant points from the documents.\n3. Conciseness: The answer is cluttered with repeated names and formatting errors that make it unnecessarily lengthy and harder to read.\n4. Avoiding unfounded responses: The answer is founded on the document but suffers from poor editing issues.\n5. Specificity: The answer includes specific pet peeves, but it also includes unnecessary repetition.\n6. Tip Inclusion: The query did not ask for tips, so this criterion is not strictly applicable.\n\nFinal result: Bad\n\nThe repeated names and formatting errors indicate lack of editing and care in the response, making it difficult to read and comprehend fully. It does not meet the required standard for a good, concise, and clear response.'
"Reasoning:\n\n1. Context Grounding: The answer only indirectly relates to the question. The provided information gives various ways to interact with the company's systems or tools rather than specifically how to access the training resources.\n2. Relevance: The answer deviates from the main question. It mentions systems for personal documents, expense claims, password management, secure sharing, feedback processes, and learning budgets, none of which directly answer how to access training resources.\n3. Conciseness: While the answer is detailed, it includes unnecessary information that isn't directly related to accessing training resources, making it not concise.\n4. Specificity: The answer lacks details on accessing training materials or platforms where these resources are housed.\n5. Generalization: It provides general guidance on several aspects but fails to narrow down specifically on training resources or offer relevant tips directly tied to them.\n\nFinal result: Bad"

'Reasoning:\n\n1. Context Grounding: The answer references various elements from the documents provided, such as giving feedback at the time of the event, focusing on the situation and not the person, and showing appreciation. It generally aligns with the tips described in the documents.\n \n2. Relevance: The answer directly addresses the question of how feedback should be given, touching on many of the specific points mentioned in the documents.\n\n3. Conciseness: The response is overly wordy and includes some repetition (e.g., focusing on the situation not the person and the intention behind giving feedback). It can be more concise while still covering the necessary points.\n\n4. Document Dependency: The response accurately reflects the content of the provided documents. However, it brings in names like "Christopher Estes," "emichelle James Johnson MD," which may be elements of the document's placeholders rather than relevant names in the context of feedback tips. This adds unnecessary confusion to the answer.\n\n5. Specificity: The answer appropriately includes specific advice, though it overdoes some parts (like the role and intentions in giving feedback).\n\n6. Accuracy and Clarity: There are minor grammatical errors and some parts like "emichelle James Johnson MDamples" which are not clear and probably are typos or formatting errors.\n\nGiven the evaluation above:\n\nFinal Result: Bad'
'Reasoning:\n1. Context Grounding: The answer refers to sharing information from high-level meetings which is supported by Document 4. However, the inclusion of "Robin Nichols" appears to be an error or typo, which diminishes the context grounding slightly.\n2. Relevance: The answer is relevant to the question asked; it covers the need for transparency, team alignment, addressing concerns, and fostering an open environment, which is in line with the information in Document 4.\n3. Conciseness: The answer is moderately concise but slightly verbose. It can be trimmed down while still maintaining the key points.\n4. Directness: The answer attempts to respond to the question based on the content provided in Document 4.\n5. Specificity: The answer includes specific points about decisions made and the rationale being shared, aligning the team and fostering a sense of purpose.\n6. Tips: The answer implicitly includes tips such as ensuring transparency and addressing team concerns.\n\nFinal Result: Good'
'Reasoning:\n\n1. Context Grounding: The given answer closely follows the instructions provided in Document 1 regarding car travel expenses. The essential details, such as keeping track of kilometers and submitting an Excel document or email to specified email addresses, are accurately mentioned.\n \n2. Relevance: The answer is pertinent to the question and provides specific steps on how to report car travel expenses, which is exactly what was asked.\n\n3. Conciseness: The response is generally concise, though it includes a minor unnecessary detail about requesting a parking card for a specific date which doesn’t directly pertain to the monthly reporting of car travel expenses.\n\n4. Specificity: The answer effectively covers the key points mentioned in the document for reporting travel expenses, including the specific reimbursement rate.\n\n5. General Tips: The tip about requesting a parking card is accurate but not directly relevant to the primary question of how to report car travel expenses.\n\nGiven the accurate detailing and mostly relevant content, while removing the unnecessary detail about the parking card date would improve the response, the answer is still sufficiently useful.\n\nFinal Result: Good'

Evaluation

Metrics

Label	Accuracy
all	0.6716

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("Netta1994/setfit_baai_newrelic_gpt-4o_cot-instructions_only_reasoning_1726750606.384621")
# Run inference
preds = model("Reasoning:
1. Context Grounding: The response draws from the documents providing relevant sources such as the organization's website, job ads, and newsletter link.
2. Relevance: The answer is directly related to the question about understanding the organization's products, challenges, and future.
3. Conciseness: The answer is clear and to the point.
4. Does not attempt to respond when the document lacks information: It addresses the question appropriately with the available information.
5. Specificity: The answer is specific and provides concrete steps to follow.
6. Relevant tips: The answer includes actionable steps like visiting the website, viewing job ads, and signing up for a newsletter, which are relevant.

The answer precisely matches all the criteria set for evaluation.

Final Result: Good")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	87	141.3077	245

Label	Training Sample Count
0	32
1	33

Training Hyperparameters

batch_size: (16, 16)
num_epochs: (5, 5)
max_steps: -1
sampling_strategy: oversampling
num_iterations: 20
body_learning_rate: (2e-05, 2e-05)
head_learning_rate: 2e-05
loss: CosineSimilarityLoss
distance_metric: cosine_distance
margin: 0.25
end_to_end: False
use_amp: False
warmup_proportion: 0.1
l2_weight: 0.01
seed: 42
eval_max_steps: -1
load_best_model_at_end: False

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0061	1	0.2339	-
0.3067	50	0.2693	-
0.6135	100	0.2364	-
0.9202	150	0.0942	-
1.2270	200	0.0031	-
1.5337	250	0.0019	-
1.8405	300	0.0016	-
2.1472	350	0.0016	-
2.4540	400	0.0015	-
2.7607	450	0.0013	-
3.0675	500	0.0013	-
3.3742	550	0.0012	-
3.6810	600	0.0012	-
3.9877	650	0.0012	-
4.2945	700	0.0012	-
4.6012	750	0.0011	-
4.9080	800	0.0011	-

Framework Versions

Python: 3.10.14
SetFit: 1.1.0
Sentence Transformers: 3.1.0
Transformers: 4.44.0
PyTorch: 2.4.1+cu121
Datasets: 2.19.2
Tokenizers: 0.19.1

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}