OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems

By Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Z. Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang and 4 more. Annual Meeting of the Association for Computational Linguistics, 2024

1,199 citations recorded by the source corpus.

Recent advancements have seen Large Language Models (LLMs) and Large Multimodal Models (LMMs) surpassing general human capabilities in various tasks, approaching the proficiency level of human experts across multiple domains. With traditional benchmarks becoming less challenging for these models, new rigorous challenges are essential to gauge their advanced abilities. In this work, we present OlympiadBench, an Olympiad-level bilingual multimodal scientific benchmark, featuring 8,476 problems from Olympiad-level mathematics and physics competitions, including the Chinese college entrance exam. Each problem is detailed with expert-level annotations for step-by-step reasoning. Evaluating top-tier models on OlympiadBench, we implement a comprehensive assessment methodology to accurately evaluate model responses. Notably, the best-performing model, GPT-4V, attains an average score of 17.97% on OlympiadBench, with a mere 10.74% in physics, highlighting the benchmark rigor and the intricacy of physical reasoning. Our analysis orienting GPT-4V points out prevalent issues with hallucinations, knowledge omissions, and logical fallacies. We hope that our challenging benchmark can serve as a valuable resource for helping future AGI research endeavors. The data and evaluation code are available at \url{https://github.com/OpenBMB/OlympiadBench}

View Paper: https://www.semanticscholar.org/paper/bcf2c7e3f4ed64c8294c35a59220a26dd4f40060

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. An Yang
2. Zhenru Zhang
3. Beichen Zhang
4. Binyuan Hui
5. Jingren Zhou
6. Runji Lin
7. Mingfeng Xue
8. Bofei Gao

Citations in the corpus

• The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models (2025, unknown): “For math tasks, the evaluation datasets include MATH500 (Hendrycks et al., 2021), AIME 2024 (Li et al., 2024), AMC (Li et al., 2024), OlympiadBench (He et al., 2024), and OMNI-MATH (Gao et al., 2024).”
• Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, unknown): “MMMU [289] and MMMU-Pro [290] are multidisciplinary reasoning benchmarks, while MathVista [163], MATH-Vision [245], MathVerse [299], and OlympiadBench [80] are mathematics benchmarks.”
• Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning (2025, unknown): “…bonus is commonly added to the training loss to encourage exploration by increasing the entropy of actions—a well-established practice in traditional tasks (Schulman et al., 2017; Williams, 1992; Mnih et al., 2016), and recently applied to LLM reasoning (Sheng et al., 2024; Hu et al., 2024).”
• Process Reinforcement through Implicit Rewards (2025, unknown): “…on competition-level mathematics and programming tasks, including AIME 2024 (Li et al., 2024), AMC (Li et al., 2024), MATH-500 (Hendrycks et al., 2021b), Minerva Math (Lewkowycz et al., 2022), Olympiad-Bench (He et al., 2024), LeetCode (Guo et al., 2024), and LiveCodeBench (v2) (Jain et al., 2024).”
• LIMO: Less is More for Reasoning (2025, unknown): “For larger benchmarks (MATH500, OlympiadBench, Gaokao, Kaoyan, GradeSchool, MinervaMath, and GPQA), we employ greedy decoding with a single sample.”
• InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency (2025, unknown): “…Hal-lusionBench [41], MVBench [63], VideoMME [35], MLVU [186], (2) Reasoning Tasks : MMMU [170], MathVista [76], MathVision [76], MathVerse [175], DynaMath [189], WeMath [100], OlympiadBench [43], LogicVista [153]; (3) Text-Centric Tasks : MATH500 [65], AIME24 [84], AIME25 [85], GPQA [106], [31].”
• The Lessons of Developing Process Reward Models in Mathematical Reasoning (2025, unknown): “5-Math-7B-Instruct and the benchmark P ROCESS B ENCH (Zheng et al., 2024) across multiple PRMs (see Table 6 and Table 7 for details).”
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (2025, unknown): “We compare the pass@ k curves of base and zero-RL models on benchmarks of varying difficulty: GSM8K (Cobbe et al., 2021), MATH500 (Hendrycks et al., 2021), Minerva (Lewkowycz et al., 2022), Olympiad (He et al., 2024), AIME24, and AMC23.”
• A benchmark of expert-level academic questions to assess AI capabilities (2025, unknown): “HLE offers a high-fidelity signal for assessing model capabilities and is publicly released to support more informed research and policy decisions.”
• rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking (2025, unknown): “…the training data is even scarcer in this regard: accurate step-by-step feedback requires intense human labeling efforts and is impractical to scale, while those automatic annotation attempts show limited gains due to noisy reward scores [Luo et al., 2024, Wang et al., 2024c, Chen et al., 2024].”
• Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement (2024, background): “These evaluation datasets include GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021b), Minerva Math (Lewkowycz et al., 2022b), Gaokao 2023 En (Liao et al., 2024), Olympiad Bench (He et al., 2024), College Math (Tang et al., 2024b), MMLU STEM (Hendrycks et al., 2021a), GaoKao (Zhong et al., 20”
• L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning (2025, unknown): “L1 significantly outperforms other length-controlled models while maintaining strong performance.”
• Are We Measuring Strategy or Phrasing? The Gap Between Surface- and Approach-Level Diversity in LLM Math Reasoning (2026, unknown): “Table 12 shows that the Approach checkpoint outperforms the Format checkpoint on both OlympiadBench and AIME, suggesting that approach-seeking behavior can improve accuracy in the sequential-solution setting. lems with multiple approaches, and within this sub-set improvements grow with difficulty.”