Description
Solver AI TR-3A-01 model is a tunned LLM model with 7.9B parameters, designed to help marketers with their text content generation. Built on the open-source Mistral 7B instruction-tuned model and trained on domain-specific data, the current version is focused on writing marketing text for social posts and ads. This is part of our exploration into leveraging AI to create effective, engaging social media content that is targeted and on-brand.
Usage
Below we share some code snippets on how to get quickly started with running the model. First, update the Transformers library with:
pip install -U transformers
Running the model on GPU
# pip install accelerate
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
tokenizer = AutoTokenizer.from_pretrained("solverai/TR-3A-01")
model = AutoModelForCausalLM.from_pretrained(
"solverai/TR-3A-01",
device_map="auto",
torch_dtype=torch.bfloat16,
)
messages = [
{"role": "user", "content": "create a social post for buying donuts in Twiter style."},
]
input_ids = tokenizer.apply_chat_template(messages, return_tensors="pt", return_dict=True).to("cuda")
outputs = model.generate(**input_ids, max_new_tokens=256)
print(tokenizer.decode(outputs[0]))
Benchmarks
To understand the capabilities, we use a variety of benchmarks. At the high-level overview of the model quality on representative benchmarks:
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| arc_challenge | 1 | none | 0 | acc | ↑ | 0.4838 | ± | 0.0146 |
| none | 0 | acc_norm | ↑ | 0.5102 | ± | 0.0146 | ||
| gsm8k | 3 | flexible-extract | 5 | exact_match | ↑ | 0.4572 | ± | 0.0137 |
| strict-match | 5 | exact_match | ↑ | 0.4428 | ± | 0.0137 | ||
| hellaswag | 1 | none | 0 | acc | ↑ | 0.6082 | ± | 0.0049 |
| none | 0 | acc_norm | ↑ | 0.7946 | ± | 0.0040 | ||
| mmlu | 1 | none | acc | ↑ | 0.5664 | ± | 0.0039 | |
| - humanities | 1 | none | acc | ↑ | 0.5105 | ± | 0.0068 | |
| - formal_logic | 0 | none | 0 | acc | ↑ | 0.4524 | ± | 0.0445 |
| - high_school_european_history | 0 | none | 0 | acc | ↑ | 0.6788 | ± | 0.0365 |
| - high_school_us_history | 0 | none | 0 | acc | ↑ | 0.7745 | ± | 0.0293 |
| - high_school_world_history | 0 | none | 0 | acc | ↑ | 0.7553 | ± | 0.0280 |
| - international_law | 0 | none | 0 | acc | ↑ | 0.7438 | ± | 0.0398 |
| - jurisprudence | 0 | none | 0 | acc | ↑ | 0.7130 | ± | 0.0437 |
| - logical_fallacies | 0 | none | 0 | acc | ↑ | 0.7607 | ± | 0.0335 |
| - moral_disputes | 0 | none | 0 | acc | ↑ | 0.6098 | ± | 0.0263 |
| - moral_scenarios | 0 | none | 0 | acc | ↑ | 0.2782 | ± | 0.0150 |
| - philosophy | 0 | none | 0 | acc | ↑ | 0.6431 | ± | 0.0272 |
| - prehistory | 0 | none | 0 | acc | ↑ | 0.6173 | ± | 0.0270 |
| - professional_law | 0 | none | 0 | acc | ↑ | 0.3950 | ± | 0.0125 |
| - world_religions | 0 | none | 0 | acc | ↑ | 0.8129 | ± | 0.0299 |
| - other | 1 | none | acc | ↑ | 0.6379 | ± | 0.0083 | |
| - business_ethics | 0 | none | 0 | acc | ↑ | 0.5100 | ± | 0.0502 |
| - clinical_knowledge | 0 | none | 0 | acc | ↑ | 0.6491 | ± | 0.0294 |
| - college_medicine | 0 | none | 0 | acc | ↑ | 0.5376 | ± | 0.0380 |
| - global_facts | 0 | none | 0 | acc | ↑ | 0.3000 | ± | 0.0461 |
| - human_aging | 0 | none | 0 | acc | ↑ | 0.6099 | ± | 0.0327 |
| - management | 0 | none | 0 | acc | ↑ | 0.7476 | ± | 0.0430 |
| - marketing | 0 | none | 0 | acc | ↑ | 0.8547 | ± | 0.0231 |
| - medical_genetics | 0 | none | 0 | acc | ↑ | 0.6400 | ± | 0.0482 |
| - miscellaneous | 0 | none | 0 | acc | ↑ | 0.7497 | ± | 0.0155 |
| - nutrition | 0 | none | 0 | acc | ↑ | 0.6699 | ± | 0.0269 |
| - professional_accounting | 0 | none | 0 | acc | ↑ | 0.4326 | ± | 0.0296 |
| - professional_medicine | 0 | none | 0 | acc | ↑ | 0.6324 | ± | 0.0293 |
| - virology | 0 | none | 0 | acc | ↑ | 0.4398 | ± | 0.0386 |
| - social sciences | 1 | none | acc | ↑ | 0.6727 | ± | 0.0082 | |
| - econometrics | 0 | none | 0 | acc | ↑ | 0.3772 | ± | 0.0456 |
| - high_school_geography | 0 | none | 0 | acc | ↑ | 0.7172 | ± | 0.0321 |
| - high_school_government_and_politics | 0 | none | 0 | acc | ↑ | 0.7979 | ± | 0.0290 |
| - high_school_macroeconomics | 0 | none | 0 | acc | ↑ | 0.5846 | ± | 0.0250 |
| - high_school_microeconomics | 0 | none | 0 | acc | ↑ | 0.6597 | ± | 0.0308 |
| - high_school_psychology | 0 | none | 0 | acc | ↑ | 0.7982 | ± | 0.0172 |
| - human_sexuality | 0 | none | 0 | acc | ↑ | 0.6412 | ± | 0.0421 |
| - professional_psychology | 0 | none | 0 | acc | ↑ | 0.5637 | ± | 0.0201 |
| - public_relations | 0 | none | 0 | acc | ↑ | 0.6182 | ± | 0.0465 |
| - security_studies | 0 | none | 0 | acc | ↑ | 0.6857 | ± | 0.0297 |
| - sociology | 0 | none | 0 | acc | ↑ | 0.8408 | ± | 0.0259 |
| - us_foreign_policy | 0 | none | 0 | acc | ↑ | 0.7700 | ± | 0.0423 |
| - stem | 1 | none | acc | ↑ | 0.4757 | ± | 0.0086 | |
| - abstract_algebra | 0 | none | 0 | acc | ↑ | 0.2500 | ± | 0.0435 |
| - anatomy | 0 | none | 0 | acc | ↑ | 0.5556 | ± | 0.0429 |
| - astronomy | 0 | none | 0 | acc | ↑ | 0.6711 | ± | 0.0382 |
| - college_biology | 0 | none | 0 | acc | ↑ | 0.6389 | ± | 0.0402 |
| - college_chemistry | 0 | none | 0 | acc | ↑ | 0.3900 | ± | 0.0490 |
| - college_computer_science | 0 | none | 0 | acc | ↑ | 0.5200 | ± | 0.0502 |
| - college_mathematics | 0 | none | 0 | acc | ↑ | 0.3700 | ± | 0.0485 |
| - college_physics | 0 | none | 0 | acc | ↑ | 0.4216 | ± | 0.0491 |
| - computer_security | 0 | none | 0 | acc | ↑ | 0.7000 | ± | 0.0461 |
| - conceptual_physics | 0 | none | 0 | acc | ↑ | 0.4298 | ± | 0.0324 |
| - electrical_engineering | 0 | none | 0 | acc | ↑ | 0.4759 | ± | 0.0416 |
| - elementary_mathematics | 0 | none | 0 | acc | ↑ | 0.3704 | ± | 0.0249 |
| - high_school_biology | 0 | none | 0 | acc | ↑ | 0.6839 | ± | 0.0265 |
| - high_school_chemistry | 0 | none | 0 | acc | ↑ | 0.4975 | ± | 0.0352 |
| - high_school_computer_science | 0 | none | 0 | acc | ↑ | 0.6200 | ± | 0.0488 |
| - high_school_mathematics | 0 | none | 0 | acc | ↑ | 0.2815 | ± | 0.0274 |
| - high_school_physics | 0 | none | 0 | acc | ↑ | 0.3113 | ± | 0.0378 |
| - high_school_statistics | 0 | none | 0 | acc | ↑ | 0.5093 | ± | 0.0341 |
| - machine_learning | 0 | none | 0 | acc | ↑ | 0.4196 | ± | 0.0468 |
| truthfulqa_mc2 | 2 | none | 0 | acc | ↑ | 0.5575 | ± | 0.0154 |
| winogrande | 1 | none | 0 | acc | ↑ | 0.7206 | ± | 0.0126 |
Sample Output
User: create a social media post for 20% off for diva sneakers, make it a instagram post
Assistant:
🌟👟 Don't miss out on this Instagram exclusive!🎉 Our Diva Sneakers
are back and we're offering a special discount just for you, my fashionistas!👊
For a limited time, if you buy two pairs, you'll receive 20% off your purchase.
👟But hurry, this offer is only available on our website, so head over there
now and grab your new kicks. Tag a friend who needs these and hit the shop!
💫#diva sneakers #limitedtimeoffer #fashionistas #backinstock
User: same thing, create a twitter post, keep it short
Assistant:
👟 Back in stock now!👊 For a limited time, buy two pairs and get 20% off on
our Instagram exclusive Diva Sneakers!🌟 Don't miss out on this limited time offer!
#diva sneakers #limitedtimeoffer #backinstock
User: get a new look for summer with Deviant Diva's new makeup lin
Assistant:
🌊Summer, sun, and style! 💆🤩 Ready to refresh your wardrobe with our latest
collection of makeup products? Get ready to take over the beauty world this
summer with Deviant Diva's hottest makeup line for the season! Shop now and
enjoy our new range of light, shimmery,and metallic shades for a flawless
all-day look. So go ahead, treat yourself and let your inner beauty shine!
🌸🌱 #beautylover #summermakeup #beautycommunity #beautyaddict
User: start your shopify store today
Assistant:
🤎🛍️ Ready to start your own online store?🛍️ With Shopify, you can create
your unique and personalized storefront and streamline your business
processes, customer service, marketing,and more! From inventory management
to shipping and fulfillment, this e-commerce platform has everything you
need to start your online shop. Sign up today and unlock your passion with
your handmade products! 🤩 #e-commerce #Shopify #SmallBusiness #StartingYourOwnBusiness
User: buy a fender strat
Assistant:
🎸🎶Get ready to unleash your potential with a Fender Stratocaster! 🎸
Whether you're playingblues, rock, or pop, this legendary guitar is perfect
for you. With its bright and clear sound,smooth fingerboard, and versatile
pickup selector, you'll sound and look like a rockstar!🤘 So go ahead and
invest in your musical journey today! 🎸 #fender stratocaster #guitarlover #musiclover #rockstar
User: check out the new allbirds
Assistant:
🌺🌳 Discover a comfortable and stylish shoe that's not only stylish but
also eco-friendly and sustainable! 🌎🌲 Allbirds' shoes are made from
Merino wool and recycled materials, ensuring fashion-conscious consumers
can enjoy a guilt-free shopping experience. From casual walking to adventurous
hiking, these shoes will take you anywhere in style! 🙌 Check out the new line and
start enjoying nature while looking fresh and cool! #allbirds #sustainableshoes
#recycledmaterials #adventuretime #naturalmaterials
User: create a post about rivian cars
Assistant:
🌊Rev up your summer with our latest collection of stylish cars from Rivian! 🌊
From eco-friendly convertibles to sleek and sporty SUVs, our luxury cars add
a touch of elegance and sophistication to your summer style. Whether you're
driving to work, heading out on adventures, or lounging by the pool, our
stylish cars give you the perfect ride with comfortability and style. Shop
now and complete your perfect summer look today! 🌊🌞 #luxurrycars #ecofriendly
#travelingmadeeasier #luxurylife #Rivianluxury
Ethical Considerations and Limitations
As we continue to develop and refine our social media post model, it's essential to acknowledge the ethical considerations and limitations inherent in its deployment. Here are some key points:
Ethical Considerations
Bias and Fairness: Our model is fine-tuned on domain datasets, but still there remains a risk of unintentional bias, particularly regarding sensitive topics like gender, race, and socioeconomic status. We are committed to continuous monitoring and updating to mitigate these biases.
Impact on Human Interaction: Our model aims to foster positive and inclusive conversations on social media. We are aware of the risks of echo chambers and polarization and strive to design the model in a way that encourages diverse perspectives and healthy dialogue.
Limitations
Contextual Understanding: The model may not always grasp the nuanced context of social media posts, leading to occasional misinterpretations or irrelevant content generation.
Cultural Sensitivity: Due to the global nature of social media, the model may sometimes lack sensitivity to cultural differences, idioms, or local norms.
Data Dependence: The model's performance is heavily influenced by the quality and diversity of its training data. We acknowledge that biases in the training data can impact the outputs and are continuously working to improve data quality.
Scalability: While the model is designed to handle large-scale content, real-time processing and resource demands can present challenges, particularly in high-traffic environments.
Our Commitment We are dedicated to ethical AI development and are continuously improving our model to better serve the community while addressing these challenges. We encourage users to provide feedback and work with us to create a safer, more inclusive, and effective tool.
Thank you for your support and understanding as we navigate the complexities of deploying AI in social media contexts.
- Downloads last month
- 0