File size: 27,397 Bytes
f8d26f9 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 |
---
base_model: BAAI/bge-base-en-v1.5
library_name: setfit
metrics:
- accuracy
pipeline_tag: text-classification
tags:
- setfit
- sentence-transformers
- text-classification
- generated_from_setfit_trainer
widget:
- text: "Reasoning:\n\n1. **Context Grounding**: The provided answer does not directly\
\ reference where to access specific training resources. It rather gives a mixture\
\ of unrelated information such as using a password manager, secure file sharing,\
\ personal development budget, etc. These points, while valid, do not directly\
\ pertain to accessing training resources specifically.\n \n2. **Relevance**:\
\ The response deviates from speaking directly about accessing training resources\
\ and instead covers a broader range of topics, which includes using company systems\
\ for other purposes, security protocols, and personal development budget discussions.\
\ These areas do not directly answer the question.\n\n3. **Conciseness**: The\
\ answer lacks conciseness as it includes quite a bit of unrelated information,\
\ which makes it unnecessarily lengthy and potentially confusing.\n\n4. **Specificity**:\
\ The answer fails to be specific about how to access training resources. If the\
\ document does contain the relevant information, it hasn't been effectively pulled\
\ out or emphasized in the answer.\n\n5. **No-Response Principle**: If a specific\
\ question cannot be answered based on the given document, it should be clearly\
\ stated. Here, it seems the document may not contain a straightforward guide\
\ to accessing training resources, but this isn't clarified.\n\nFinal Result:\
\ **Bad**"
- text: '**Reasoning:**
1. **Context Grounding:** The answer is well-supported by Document 1, which specifies
that questions about travel reimbursement should be directed to finance@ORGANIZATION_2.
2. **Relevance:** The answer is directly relevant to the question, which asks
whom to contact about travel reimbursement.
3. **Conciseness:** The answer is concise and directly addresses the question
without including any unnecessary information.
4. **Response Appropriateness:** The answer correctly utilizes the detail provided
in the document and does not attempt to provide unnecessary information.
5. **Specificity:** The answer provides the specific email address for the relevant
point of contact.
6. **Balanced Detail:** The answer includes the required contact information and
is not too general.
**Final Result: Good.**'
- text: 'The answer provided fulfills the criteria as outlined:
1. **Context Grounding**: The answer accurately reflects the contents of the documents,
particularly Document 1, which specifies the steps team leads should take when
they consider the possibility that someone''s time at the ORGANIZATION is up.
2. **Relevance**: The answer directly addresses the question by explaining why
it is important for team leads to think about the possibility of someone leaving,
including the potential benefits of addressing issues before they become unmanageable.
3. **Conciseness**: The answer is somewhat verbose but manages to stay mostly
on-topic and addresses the question throughout, ensuring it doesn''t veer into
unrelated topics.
4. **Specificity**: While a bit generalized in parts, the answer includes specifics
such as underperformance, lack of growth, and disagreement with company direction,
which are supported by the details in the documents.
Therefore, despite minor verbosity, the answer is accurate, relevant, and detailed
enough to be considered good.
**Final Result: Good**'
- text: 'Reasoning:
1. Context Grounding: The response draws from the documents providing relevant
sources such as the organization''s website, job ads, and newsletter link.
2. Relevance: The answer is directly related to the question about understanding
the organization''s products, challenges, and future.
3. Conciseness: The answer is clear and to the point.
4. Does not attempt to respond when the document lacks information: It addresses
the question appropriately with the available information.
5. Specificity: The answer is specific and provides concrete steps to follow.
6. Relevant tips: The answer includes actionable steps like visiting the website,
viewing job ads, and signing up for a newsletter, which are relevant.
The answer precisely matches all the criteria set for evaluation.
Final Result: Good'
- text: 'Reasoning:
1. Context Grounding: The answer is well-grounded in the document, explaining
the roles of ORGANIZATION_2, Thomas Barnes, and Charlotte Herrera correctly based
on the provided information.
2. Relevance: The answer directly addresses the question, detailing the extent
of ORGANIZATION_2''s participation in the farewell process.
3. Conciseness: While mostly concise, the answer could be slightly more succinct.
Repetition of certain elements could be minimized.
4. Appropriateness of Response: The answer does not attempt to respond beyond
the document.
5. Specificity: The answer is specific, describing the involvement of specific
people and their roles.
6. Tips and Generality: The answer includes relevant insights into the roles,
though it does not go into overly general territory.
Final Result: Good'
inference: true
model-index:
- name: SetFit with BAAI/bge-base-en-v1.5
results:
- task:
type: text-classification
name: Text Classification
dataset:
name: Unknown
type: unknown
split: test
metrics:
- type: accuracy
value: 0.6716417910447762
name: Accuracy
---
# SetFit with BAAI/bge-base-en-v1.5
This is a [SetFit](https://github.com/huggingface/setfit) model that can be used for Text Classification. This SetFit model uses [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) as the Sentence Transformer embedding model. A [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance is used for classification.
The model has been trained using an efficient few-shot learning technique that involves:
1. Fine-tuning a [Sentence Transformer](https://www.sbert.net) with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.
## Model Details
### Model Description
- **Model Type:** SetFit
- **Sentence Transformer body:** [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5)
- **Classification head:** a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance
- **Maximum Sequence Length:** 512 tokens
- **Number of Classes:** 2 classes
<!-- - **Training Dataset:** [Unknown](https://huggingface.co/datasets/unknown) -->
<!-- - **Language:** Unknown -->
<!-- - **License:** Unknown -->
### Model Sources
- **Repository:** [SetFit on GitHub](https://github.com/huggingface/setfit)
- **Paper:** [Efficient Few-Shot Learning Without Prompts](https://arxiv.org/abs/2209.11055)
- **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit)
### Model Labels
| Label | Examples |
|:------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 0 | <ul><li>"Reasoning: \n\n1. **Context Grounding**: The answer pulls loosely from the documents provided but does not ground the response specifically to any of the documents' key points about budgeting or investment decisions.\n2. **Relevance**: While the query is about saving money in general, the answer ambiguously addresses saving money by suggesting wise spending and seeking guidance, which are indirectly related to what's mentioned in the documents.\n3. **Conciseness**: The answer contains unnecessary details like mentioning health and priorities, which are somewhat related based on Document 3 but are not directly in line with the exact scope of the question.\n4. **Too General**: The answer is broad and generic. The document contains specific guidelines (such as asking if the investment would be made with personal money and consulting with ORGANIZATION_2 for high-stakes cases), which the response fails to integrate effectively.\n5. **Detail Inclusion**: The answer needed to incorporate specific strategies mentioned in the documents, like ensuring if an expense aligns with personal and organizational benefits or seeking guidance on higher-budget items.\n\nFinal result: **Bad**"</li><li>'**Reasoning:**\n\n1. **Context Grounding**: The answer does mention several pet peeves directly from the provided documents, though it incorrectly repeats the names and has some formatting errors, it aligns generally with the content.\n2. **Relevance**: The answer focuses on the specific question about pet peeves, summarizing relevant points from the documents.\n3. **Conciseness**: The answer is cluttered with repeated names and formatting errors that make it unnecessarily lengthy and harder to read.\n4. **Avoiding unfounded responses**: The answer is founded on the document but suffers from poor editing issues.\n5. **Specificity**: The answer includes specific pet peeves, but it also includes unnecessary repetition.\n6. **Tip Inclusion**: The query did not ask for tips, so this criterion is not strictly applicable.\n\n**Final result**: **Bad**\n\nThe repeated names and formatting errors indicate lack of editing and care in the response, making it difficult to read and comprehend fully. It does not meet the required standard for a good, concise, and clear response.'</li><li>"Reasoning:\n\n1. **Context Grounding**: The answer only indirectly relates to the question. The provided information gives various ways to interact with the company's systems or tools rather than specifically how to access the training resources.\n2. **Relevance**: The answer deviates from the main question. It mentions systems for personal documents, expense claims, password management, secure sharing, feedback processes, and learning budgets, none of which directly answer how to access training resources.\n3. **Conciseness**: While the answer is detailed, it includes unnecessary information that isn't directly related to accessing training resources, making it not concise.\n4. **Specificity**: The answer lacks details on accessing training materials or platforms where these resources are housed.\n5. **Generalization**: It provides general guidance on several aspects but fails to narrow down specifically on training resources or offer relevant tips directly tied to them.\n\nFinal result: **Bad**"</li></ul> |
| 1 | <ul><li>'**Reasoning:**\n\n1. **Context Grounding**: The answer references various elements from the documents provided, such as giving feedback at the time of the event, focusing on the situation and not the person, and showing appreciation. It generally aligns with the tips described in the documents.\n \n2. **Relevance**: The answer directly addresses the question of how feedback should be given, touching on many of the specific points mentioned in the documents.\n\n3. **Conciseness**: The response is overly wordy and includes some repetition (e.g., focusing on the situation not the person and the intention behind giving feedback). It can be more concise while still covering the necessary points.\n\n4. **Document Dependency**: The response accurately reflects the content of the provided documents. However, it brings in names like "Christopher Estes," "emichelle James Johnson MD," which may be elements of the document\'s placeholders rather than relevant names in the context of feedback tips. This adds unnecessary confusion to the answer.\n\n5. **Specificity**: The answer appropriately includes specific advice, though it overdoes some parts (like the role and intentions in giving feedback).\n\n6. **Accuracy and Clarity**: There are minor grammatical errors and some parts like "emichelle James Johnson MDamples" which are not clear and probably are typos or formatting errors.\n\nGiven the evaluation above:\n\n**Final Result: Bad**'</li><li>'Reasoning:\n1. **Context Grounding:** The answer refers to sharing information from high-level meetings which is supported by Document 4. However, the inclusion of "Robin Nichols" appears to be an error or typo, which diminishes the context grounding slightly.\n2. **Relevance:** The answer is relevant to the question asked; it covers the need for transparency, team alignment, addressing concerns, and fostering an open environment, which is in line with the information in Document 4.\n3. **Conciseness:** The answer is moderately concise but slightly verbose. It can be trimmed down while still maintaining the key points.\n4. **Directness:** The answer attempts to respond to the question based on the content provided in Document 4.\n5. **Specificity:** The answer includes specific points about decisions made and the rationale being shared, aligning the team and fostering a sense of purpose.\n6. **Tips:** The answer implicitly includes tips such as ensuring transparency and addressing team concerns.\n\nFinal Result: Good'</li><li>'**Reasoning:**\n\n1. **Context Grounding:** The given answer closely follows the instructions provided in Document 1 regarding car travel expenses. The essential details, such as keeping track of kilometers and submitting an Excel document or email to specified email addresses, are accurately mentioned.\n \n2. **Relevance:** The answer is pertinent to the question and provides specific steps on how to report car travel expenses, which is exactly what was asked.\n\n3. **Conciseness:** The response is generally concise, though it includes a minor unnecessary detail about requesting a parking card for a specific date which doesn’t directly pertain to the monthly reporting of car travel expenses.\n\n4. **Specificity:** The answer effectively covers the key points mentioned in the document for reporting travel expenses, including the specific reimbursement rate.\n\n5. **General Tips:** The tip about requesting a parking card is accurate but not directly relevant to the primary question of how to report car travel expenses.\n\nGiven the accurate detailing and mostly relevant content, while removing the unnecessary detail about the parking card date would improve the response, the answer is still sufficiently useful.\n\n**Final Result:** Good'</li></ul> |
## Evaluation
### Metrics
| Label | Accuracy |
|:--------|:---------|
| **all** | 0.6716 |
## Uses
### Direct Use for Inference
First install the SetFit library:
```bash
pip install setfit
```
Then you can load this model and run inference.
```python
from setfit import SetFitModel
# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("Netta1994/setfit_baai_newrelic_gpt-4o_cot-instructions_only_reasoning_1726750606.384621")
# Run inference
preds = model("Reasoning:
1. Context Grounding: The response draws from the documents providing relevant sources such as the organization's website, job ads, and newsletter link.
2. Relevance: The answer is directly related to the question about understanding the organization's products, challenges, and future.
3. Conciseness: The answer is clear and to the point.
4. Does not attempt to respond when the document lacks information: It addresses the question appropriately with the available information.
5. Specificity: The answer is specific and provides concrete steps to follow.
6. Relevant tips: The answer includes actionable steps like visiting the website, viewing job ads, and signing up for a newsletter, which are relevant.
The answer precisely matches all the criteria set for evaluation.
Final Result: Good")
```
<!--
### Downstream Use
*List how someone could finetune this model on their own dataset.*
-->
<!--
### Out-of-Scope Use
*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->
<!--
## Bias, Risks and Limitations
*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
-->
<!--
### Recommendations
*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
-->
## Training Details
### Training Set Metrics
| Training set | Min | Median | Max |
|:-------------|:----|:---------|:----|
| Word count | 87 | 141.3077 | 245 |
| Label | Training Sample Count |
|:------|:----------------------|
| 0 | 32 |
| 1 | 33 |
### Training Hyperparameters
- batch_size: (16, 16)
- num_epochs: (5, 5)
- max_steps: -1
- sampling_strategy: oversampling
- num_iterations: 20
- body_learning_rate: (2e-05, 2e-05)
- head_learning_rate: 2e-05
- loss: CosineSimilarityLoss
- distance_metric: cosine_distance
- margin: 0.25
- end_to_end: False
- use_amp: False
- warmup_proportion: 0.1
- l2_weight: 0.01
- seed: 42
- eval_max_steps: -1
- load_best_model_at_end: False
### Training Results
| Epoch | Step | Training Loss | Validation Loss |
|:------:|:----:|:-------------:|:---------------:|
| 0.0061 | 1 | 0.2339 | - |
| 0.3067 | 50 | 0.2693 | - |
| 0.6135 | 100 | 0.2364 | - |
| 0.9202 | 150 | 0.0942 | - |
| 1.2270 | 200 | 0.0031 | - |
| 1.5337 | 250 | 0.0019 | - |
| 1.8405 | 300 | 0.0016 | - |
| 2.1472 | 350 | 0.0016 | - |
| 2.4540 | 400 | 0.0015 | - |
| 2.7607 | 450 | 0.0013 | - |
| 3.0675 | 500 | 0.0013 | - |
| 3.3742 | 550 | 0.0012 | - |
| 3.6810 | 600 | 0.0012 | - |
| 3.9877 | 650 | 0.0012 | - |
| 4.2945 | 700 | 0.0012 | - |
| 4.6012 | 750 | 0.0011 | - |
| 4.9080 | 800 | 0.0011 | - |
### Framework Versions
- Python: 3.10.14
- SetFit: 1.1.0
- Sentence Transformers: 3.1.0
- Transformers: 4.44.0
- PyTorch: 2.4.1+cu121
- Datasets: 2.19.2
- Tokenizers: 0.19.1
## Citation
### BibTeX
```bibtex
@article{https://doi.org/10.48550/arxiv.2209.11055,
doi = {10.48550/ARXIV.2209.11055},
url = {https://arxiv.org/abs/2209.11055},
author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
title = {Efficient Few-Shot Learning Without Prompts},
publisher = {arXiv},
year = {2022},
copyright = {Creative Commons Attribution 4.0 International}
}
```
<!--
## Glossary
*Clearly define terms in order to be accessible across audiences.*
-->
<!--
## Model Card Authors
*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->
<!--
## Model Card Contact
*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
--> |