When will a single model first achieve 10@k solve rate >= 90% on the CodeContests dataset?
Mini
7
Ṁ3062032
1D
1W
1M
ALL
4%
5%
Before 2023
11%
2023-2023
17%
2024-2025
25%
2025-2026
18%
2026-2027
17%
2027-2032
1.6%
2032-never
10@k - unlimited samples may be generated, but at most 10 code samples may be submitted per problem.
I will post my own responses and pick whichever one of those turns out to be correct.
CodeContests dataset: https://github.com/deepmind/code_contests
Close date updated to 2050-01-01 11:59 pm
Apr 8, 4:16pm: Options are written as [a, b), so 2024-2025 means occurring after the start of 2024 and before the start of 2025.
Close date updated to 2032-01-01 11:59 pm
Get Ṁ1,000 play money
Sort by:
@ValeryCherepanov Yeah the before 2023 option seems like free money to me, I think the market is just old and people don't have it on their radar.
Related questions
Related questions
Will any model get above human level (92%) on the Simple Bench benchmark before September 1st, 2025.
60% chance
When will an AI model be better than me at competitive programming?
Will the winner of the ARC Kaggle competition solve more than 50% of the tasks
73% chance
By when will SAT solvers be 10x faster (than 2023 on same hardware)?
AI resolves at least X% on SWE-bench without any assistance, by 2028?
When will AI be able to solve 50% of official Jane Street puzzles?
Will I get 2100+ on codeforces in 2024?
67% chance
Will an AI achieve a perfect score on the Miklós Schweitzer Competition before 2035?
57% chance
Will any AI be able to formalize >=90% of IMO problems by the start of 2025?
20% chance
Benchmark Gap #3: Once a model achieves superhuman performance on a competitive programming benchmark, will it be less than 2 years before there are "entry level" AI programmers in industry use?
68% chance