What is KMMLU in the model card?
Hello, Thanks for providing such a usable model.
I am wondering what is KMMLU in the model card? i guess that should be a korean MMLU as an evaluation metric, but then why this model works little bit lower than original model? And how could you sure both "korean and english" performance is good enouogh? Can you disclose more evaluation results related to korean language generation? Or simple inference code for korean response generation.
Hi Andrea,
K-MMLU was recently incorporated into the lm_evaluation_harness. For more details, visit: K-MMLU on lm_evaluation_harness. The minor differences between this model and the base model are negligible, as they fall within the margin of error (±).
For insights into the English performance of this model, you can refer to the leaderboard here: Open LLM Leaderboard. Filter the results by "SOLAR" to see that this model's performance is almost identical to that of the base model.
For inference purposes, I recommend using vLLM. You can start quickly by following this guide: vLLM Quickstart. Change the model name to whichever you wish to test. Please note that this model is nearly identical to the base model and may not follow instructions as accurately as other instruction-tuned models, such as Bookworm-10.7B-v0.3.
Thanks,
Seungduk