Model failed CFA Level 1
I ran this on one of my internal CFA Level 1 exam tests and the model only scored 54.7%, all scores are 0 shot. It did beat the other models however.
Was this only evaluated on CFA Level 3 due to it being a written exam? Wondering how that was marked and if any other testing was done on CFA.
@mukaj Make sure to use the prompt provided below, as that is what our team used when conducting the evaluation for level 3.
Take a moment to calm your mind and focus. You are preparing for the CFA® Level III exam, which demands precision and deep understanding. Carefully read the scenario provided and the subsequent question. Your task is to analyze the scenario and select the most appropriate answer from the options given.
Scenario:
[Insert the specific scenario file input here]
Question:
Regarding the immediate tactical asset allocation demand, the behavioral bias most likely exhibited by the client is:
Answer Choices:
A. Loss aversion
B. Home bias
C. Representativeness bias
Please provide:
Your Answer: Choose A, B, or C.
Explanation: Offer a concise explanation supporting your answer, demonstrating your understanding of the behavioral biases and their implications in asset allocation.
Hi @wassemgtk ! Thanks for the great work! For CFA Level 3, there are only 8 sample questions in this website. Is this the dataset your team used to evaluate the level 3? If not, which datasets for CFA Level 3 do you evaluate on? Thanks a lot!
There are a total of 88 questions per exam. We used CFA prep providers to access past exam questions and mock exams. Unfortunately, we can't share that material publicly as we do not own the rights.
Hi @wassemgtk , We are interested in reproducing the results. Could you specify which provider and which 88 questions were used (maybe their identifiers)? We would like to have the exact same testing setup as you did, and we'll contact the provider to obtain the rights. Thanks again!