Where there are two versions? Any difference between them?
If I submit a model, where will it show up?
I think v1 is actually included in v2, isn't it?
If that is the case, why not just unify as v2?
I am confused about these three files for the current two versions as well...
Thank you very much for your interest in our work. If you submit your model results, please choose the v1 version in the "version" section, and your results will be displayed in the seed-benchmark v1. If you choose v2, they will be displayed in seed-benchmark v2. We decided to separate v1 and v2 versions because the 9-th dimension question in v2 has been expanded. In addition, the descriptions of the three JSON versions are as follows: SEED-Bench.json is the initial version we released in August; SEED-Bench-1.json is the v1 version's JSON after manually filtering the video questions; and SEED-Bench-2.json is the corresponding JSON for SEED-Bench-2.
Thank you for your attention. Since the varying performance of the same dimension across different versions might be confusing, so we have separated the leaderboards for the two versions to provide a clearer view.
Thank you for your attention. Since the varying performance of the same dimension across different versions might be confusing, so we have separated the leaderboards for the two versions to provide a clearer view.
Thanks for your quick replies.
Since v2 includes v1 and v1 is a subset of v2, why not consider using v2 only? Thus, why not merge the two leaderboards and also save a lot of effort in maintenance?
Indeed, I attempted to merge the two. However, SEED-Bench-2 has more dimensions compared to SEED-Bench-1, which could potentially lead to confusion. Therefore, we decided to keep them separate.