GPQA: A Graduate-Level Google-Proof Q&A Benchmark
By David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, Samuel R. Bowman. 2023
2,996 citations recorded by the source corpus.
We present GPQA, a challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. We ensure that the questions are high-quality and extremely difficult: experts who have or are pursuing PhDs in the corresponding domains reach 65% accuracy (74% when discounting clear mistakes the experts identified in retrospect), while highly skilled non-expert validators only reach 34% accuracy, despite spending on average over 30 minutes with unrestricted access to the web (i.e., the questions are"Google-proof"). The questions are also difficult for state-of-the-art AI systems, with our strongest GPT-4 based baseline achieving 39% accuracy. If we are to use future AI systems to help us answer very hard questions, for example, when developing new scientific knowledge, we need to develop scalable oversight methods that enable humans to supervise their outputs, which may be difficult even if the supervisors are themselves skilled and knowledgeable. The difficulty of GPQA both for skilled non-experts and frontier AI systems should enable realistic scalable oversight experiments, which we hope can help devise ways for human experts to reliably get truthful information from AI systems that surpass human capabilities.
Subfields: Artificial Intelligence, Information Systems, Evaluation & Benchmarks, Question Answering & Reading Comprehension
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Graham Neubig
- Luke S. Zettlemoyer
- Percy Liang
- Mohit Bansal
- Jianfeng Gao
- Yejin Choi
- Noah A. Smith
- Quoc V. Le
In Information Systems
- Graham Neubig
- Xiao Liu
- Yilin Niu
- Fei Huang
- Ji-rong Wen
- Mantas Mazeika
- Jingren Zhou
- Ponnurangam Kumaraguru
In Evaluation & Benchmarks
In Question Answering & Reading Comprehension
Citations in the corpus
- Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model (2025, neutral): “Future work should investigate scaling test time computation through multi-turn interactions for contextual reasoning, value models for trajectory assessment, and multi-agent scenarios for developing more sophisticated reasoning strategies.”
- Qwen3-VL Technical Report (2025, positive): “(1) Knowledge: MMLU-Pro (Wang et al., 2024f), MMLU-Redux (Gema et al., 2024), GPQA (Rein et al., 2023), Su-perGPQA (Team, 2025), (2) Reasoning: AIME-25 (AIME, 2025), HMMT-25 (HMMT, 2025), LiveBench (2024-11-25) (White et al., 2024), (3) Code: LiveCodeBench v6 (Jain et al., 2024), CFEval, OJBench (Wa”
- WebThinker: Empowering Large Reasoning Models with Deep Research Capability (2025, positive): “We utilize several benchmarks: GPQA [42], a PhD-level science question answering dataset covering physics, chemistry, and biology (we use the diamond set comprising 198 questions); GAIA [33], a benchmark designed to evaluate general AI assistants on complex information retrieval tasks (we use 103…”
- Search-o1: Agentic Search-Enhanced Large Reasoning Models (2025, positive): “Example #1 from GPQA Dataset Question: trans-cinnamaldehyde was treated with methylmagnesium bromide, forming product 1.”
- GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models (2025, neutral): “5-Air on seven benchmarks, including MMLU-Pro [43], AIME 24, MATH 500 [14], SciCode [36], GPQA [30], Humanity’s Last Exam (HLE) [28], and LiveCodeBench (LCB) [19] 2 .”
- Kimi K2: Open Agentic Intelligence (2025, neutral): “In reasoning, we include a wide range of mathematical, science and logical tasks: AIME 2024/2025, MATH-500, HMMT 2025, CNMO 2024, PolyMath-en, ZebraLogic [44], AutoLogi [92], GPQA-Diamond [62], SuperGPQA [14], and Humanity’s Last Exam (Text-Only) [57].”
- InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency (2025, neutral): “…Hal-lusionBench [41], MVBench [63], VideoMME [35], MLVU [186], (2) Reasoning Tasks : MMMU [170], MathVista [76], MathVision [76], MathVerse [175], DynaMath [189], WeMath [100], OlympiadBench [43], LogicVista [153]; (3) Text-Centric Tasks : MATH500 [65], AIME24 [84], AIME25 [85], GPQA [106], [31].”
- The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning (2024, methodology): “Multiple-choice is a common paradigm to test knowledge in language models (Hendrycks et al., 2020b; Rein et al., 2023).”
- VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning (2025, neutral): “More recent advances, including Llava-OV [Li et al., 2024], Infinity-MM [Gu et al., 2024], MAmmoTH-VL [Guo et al., 2024], and VisualWebInstruct [Jia et al., 2025], show that scaling up instruction tuning datasets and introducing diverse tasks can significantly enhance generalization across a wide…”
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (2025, positive): “• Knowledge : On benchmarks such as MMLU, MMLU-Pro, and GPQA Diamond, DeepSeek-R1 achieves outstanding results, significantly outperforming DeepSeek-V3 with scores of 90.8% on MMLU, 84.0% on MMLU-Pro, and 71.5% on GPQA Diamond.”
- Dream 7B: Diffusion Large Language Models (2025, positive): “For mathematical and scientific reasoning, we used GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2020) for numerical computation and problem-solving, and GPQA (Rein et al., 2023) for scientific understanding.”
- L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning (2025, positive): “We evaluate our models on test sets of 4 different reasoning datasets: AIME 2025, MATH (Hendrycks et al., 2021b), AMC, Olympiad-Bench (He et al., 2024), and additionally GPQA (Rein et al., 2023), LSAT (Zhong et al., 2023), and MMLU (Hendrycks et al., 2021a).”
- Reasoning Models Don't Always Say What They Think (2025, neutral): “We observe that CoTs on GPQA are consistently less faithful than those on MMLU across 5 out of 6 bias categories (the other being a tie of 0% faithfulness) for Claude 3.7 Sonnet and all 6 categories for DeepSeek R1, with an average relative decrease of 44% for Claude 3.7 Sonnet and 32% for DeepSeek ”