based on https://lmarena.ai/?leaderboard
Resolves YES if the open model with the highest Elo at EOY is from a Chinese lab.
Update 2025-03-16 (PST) (AI summary of creator comment): Resolution Clarification:
Method: The outcome will be determined using lmarena's designation.
Definition: An LLM is considered open if it is listed as something other than 'proprietary'.
Update 2025-09-08 (PST) (AI summary of creator comment): - Leaderboard mode: The highest Elo at EOY will be determined using lmarena’s default leaderboard method at that time (e.g., style-control, no-control, or any future default).
@ZandaZhu This is the market for open LLMs, which have for a while only had Chinese teams release models with near-frontier capabilities.
@ZandaZhu The best non-Chinese open LLMs are at 44th and 46th place. There are 16 Chinese open LLMs above that.
@Bayesian I'll use the default method on the leaderboard, whether it's style-control, no-control, or a different thing.
This is the correct decision because we want markets on high signal measurements, and I trust the Chatbot Arena people to continue to update ranking methods to combat sloptimisation while potentially acknowledging that style is worth something.
I bet yes because outside of China, no one really has an incentive to open source a frontier model. Chinese labs are open sourcing their near-frontier models as they compete for state support.
Meta is the only Western company with frontier level model capabilities and a strong incentive to open source. And, it seems unlikely that they will just happen to release Llama 5 in December and it will also be the best model.
@DanHomerick the plan is just to use lmarena's designation. If it's listed as something other than "proprietary" will probably count.
@Bayesian ya I had the same reaction... I still think there's less than 40% chance of this by EOY tho, tbh