TheoremQA: A Theorem-driven Question Answering dataset
By Wenhu Chen, Ming Yin, Max W.F. Ku, Yixin Wan, Xueguang Ma, Jianyu Xu, Tony Xia, Xinyi Wang, Pan Lu. 2023
250 citations recorded by the source corpus.
The recent LLMs like GPT-4 and PaLM-2 have made tremendous progress in solving fundamental math problems like GSM8K by achieving over 90% accuracy. However, their capabilities to solve more challenging math problems which require domain-specific knowledge (i.e. theorem) have yet to be investigated. In this paper, we introduce TheoremQA, the first theorem-driven question-answering dataset designed to evaluate AI models' capabilities to apply theorems to solve challenging science problems. TheoremQA is curated by domain experts containing 800 high-quality questions covering 350 theorems (e.g. Taylor's theorem, Lagrange's theorem, Huffman coding, Quantum Theorem, Elasticity Theorem, etc) from Math, Physics, EE&CS, and Finance. We evaluate a wide spectrum of 16 large language and code models with different prompting strategies like Chain-of-Thoughts and Program-of-Thoughts. We found that GPT-4's capabilities to solve these problems are unparalleled, achieving an accuracy of 51% with Program-of-Thoughts Prompting. All the existing open-sourced models are below 15%, barely surpassing the random-guess baseline. Given the diversity and broad coverage of TheoremQA, we believe it can be used as a better benchmark to evaluate LLMs' capabilities to solve challenging science problems. The data and code are released in https://github.com/wenhuchen/TheoremQA.
Subfields: Artificial Intelligence, Information Systems, Question Answering & Reading Comprehension, Automated Theorem Proving & Formal Methods
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Zhiyuan Liu
- Mohit Bansal
- Maosong Sun
- Song-Chun Zhu
- Percy Liang
- Luke S. Zettlemoyer
- Yejin Choi
In Information Systems
In Question Answering & Reading Comprehension
In Automated Theorem Proving & Formal Methods
Citations in the corpus
- MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (2024, methodology): “We then expand our dataset by incorporating questions from the STEM Website 2 , TheoremQA [10], and SciBench [40].”
- OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems (2024, background): “TheoremQA (Chen et al., 2023) is the first theorem-driven question-answering dataset.”
- Large Language Models for Mathematical Reasoning: Progresses and Challenges (2024, methodology): “…have quality concern AQ U A (Ling et al., 2017) 100K C GRE&GMAT questions A RB (Sawada et al., 2023) 105 C Contest problems and university math proof G HOSTS (Frieder et al., 2023) 709 C T HEOREM QA-M ATH (Chen et al., 2023b) 442 C Theorem as rationale LILA (Mishra et al., (Wei et al., 2022).”
- Qwen2 Technical Report (2024, unknown): “…for evaluation include MMLU (Hendrycks et al., 2021a) (5-shot), MMLU-Pro (Wang et al., 2024) (5-shot), GPQA (Rein et al., 2023) (5shot), Theorem QA (Chen et al., 2023a) (5-shot), BBH (Suzgun et al., 2023) (3-shot), HellaSwag (Zellers et al., 2019) (10-shot), Winogrande (Sak-aguchi et al., 2021)…”
- MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback (2023, unknown): “An example from TheoremQA is as follows: Task: Let M be the inverse of the group element ((3, 5), (4, 6)) in Z 7.”
- Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models (2023, background): “To address these shortcomings, there has been a growing interest in exploring the use of external tools and modular approaches to augment LLMs.”
- Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, unknown): “(3) TheoremQA [33] introduces 800 STEM-focused problems requiring theorem application in fields like mathematics, physics, and finance, with 0-shot results reported.”
- Training Large Language Models to Reason in a Continuous Latent Space (2024, unknown): “This serves as a simplified simulation of more advanced reasoning tasks, such as automated theorem proving (Chen et al., 2023; DeepMind, 2024).”
- MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning (2023, methodology): “Type Annotation # Samples Characteristics Fields GSM8K (Cobbe et al., 2021) CoT Human 7K Grade Schol Exam ∎ GSM8K-RFT (Yuan et al., 2023) CoT Llama 28K Llama + Validated ∎ AQuA-RAT (Ling et al., 2017) CoT Human 90K GRE/GMAT Exam ∎ MATH (Hendrycks et al., 2021b) CoT Human 7K Math Competition ∎ ∎ ∎ ∎ ”
- MAmmoTH2: Scaling Instructions from the Web (2024, unknown): “…in reasoning abilities across different domains, we utilize several widely used datasets, GSM8K [Cobbe et al., 2021], MATH [Hendrycks et al., 2021b], TheoremQA [Chen et al., 2023b], BIG-Bench Hard (BBH) [Suzgun et al., 2022], ARC-C [Clark [Rein et al., 2023], MMLU-STEM [Hendrycks et al., 2021a].”
- A Survey of Deep Learning for Mathematical Reasoning (2022, background): “Others, instead, present large-scale unified benchmarks for mathematical reasoning (Mishra et al., 2022b,a; Chen et al., 2023).”
- SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models (2023, methodology): “TheroemQA [8] is a theorem-oriented dataset comprising 800 high-quality questions that aim to evaluate the ability of LLMs to apply theorems to solve problems.”
- InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models (2025, unknown): “…mathematics, and coding tasks, such as MMLU [46], CMMLU [63], C-Eval [48], GAOKAO-Bench [149], TriviaQA [52], NaturalQuestions [58, 110], RACE [59], WinoGrande [103], HellaSwag [142], BigBench Hard [112], GSM8K-Test [25], MATH [47], TheoremQA [17], HumanEval [14], MBPP [4], and MBPP-CN [4].”
- Qwen2.5-Coder Technical Report (2024, methodology): “5-Coder’s mathematical performance, we selected five popular benchmarks, including MATH (Hendrycks et al., 2021), GSM8K (Cobbe et al., 2021), MMLU-STEM (Hendrycks et al., 2020) and TheoremQA (Chen et al., 2023).”