Measuring Mathematical Problem Solving With the MATH Dataset

By Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, Jacob Steinhardt. 2021

5,902 citations recorded by the source corpus.

Many intellectual endeavors require mathematical problem solving, but this skill remains beyond the capabilities of computers. To measure this ability in machine learning models, we introduce MATH, a new dataset of 12,500 challenging competition mathematics problems. Each problem in MATH has a full step-by-step solution which can be used to teach models to generate answer derivations and explanations. To facilitate future research and increase accuracy on MATH, we also contribute a large auxiliary pretraining dataset which helps teach models the fundamentals of mathematics. Even though we are able to increase accuracy on MATH, our results show that accuracy remains relatively low, even with enormous Transformer models. Moreover, we find that simply increasing budgets and model parameter counts will be impractical for achieving strong mathematical reasoning if scaling trends continue. While scaling Transformers is automatically solving most other text-based tasks, scaling is not currently solving MATH. To have more traction on mathematical problem solving we will likely need new algorithmic advancements from the broader research community.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “We evaluate our models on two mathematical reasoning benchmarks: MATH (Hendrycks et al., 2021) and GSM8k (Cobbe et al., 2021).”
Mistral 7B (2023, neutral): “…NaturalQuestions [16], TriviaQA [15] • Reading Comprehension (0-shot): BoolQ [8], QuAC [7] • Math: GSM8K [10] (8-shot) with maj@8 and MATH [13] (4-shot) with maj@4 • Code: Humaneval [5] (0-shot) and MBPP [2] (3-shot) • Popular aggregated results: MMLU [12] (5-shot), BBH [23] (3-shot), and…”
Mixtral of Experts (2024, neutral): “…(5-shot): NaturalQuestions [20], TriviaQA [19] • Reading Comprehension (0-shot): BoolQ [7], QuAC [5] • Math: GSM8K [9] (8-shot) with maj@8 and MATH [17] (4-shot) with maj@4 • Code: Humaneval [4] (0-shot) and MBPP [1] (3-shot) • Popular aggregated results: MMLU [16] (5-shot), BBH [29] (3-shot), and…”
Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, neutral): “Though simple for humans, arithmetic reasoning is a task where language models often struggle (Hendrycks et al., 2021; Patel et al., 2021, inter alia ).”
TruthfulQA: Measuring How Models Mimic Human Falsehoods (2021, background): “While most benchmarks are multiple choice, some require models to generate short (single-phrase) answers (Hendrycks et al., 2021; Lewis et al., 2020).”
Solving Quantitative Reasoning Problems with Language Models (2022, methodology): “However, these models have struggled with tasks that require quantitative reasoning, such as solving mathematics, science, and engineering problems (Hendrycks et al., 2021; Cobbe et al., 2021).”
Training Verifiers to Solve Math Word Problems (2021, methodology): “However, even the largest models falter when required to perform multi-step mathematical reasoning (Hendrycks et al., 2021).”
OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization (2022, methodology): “Furthermore, adding 1% reasoning data results in the largest gains overall, beyond which, the gains start to reduce on MMLU, Cause-Effect Accuracy, Toxicity, and Dialogue (averaged over 0 and 5-shot).”
GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, methodology): “Hendrycks et al. (2021b) find that few-shot evaluation does not improve performance, but that appears to be only the case for GPT-3.”
Galactica: A Large Language Model for Science (2022, methodology): “For KhanProblems, we used the problems from AMPS and converted to a <work> format (Hendrycks et al., 2021).”
Llemma: An Open Language Model For Mathematics (2023, methodology): “• MMLU-STEM (Hendrycks et al., 2021a), a subset of 18 out of 57 subjects in the MMLU benchmark.”
Teaching Autoregressive Language Models Complex Tasks By Demonstration (2021, background): “…“demonstrations” generated by the model provide only a limited sort of interpretability, and should not be treated as true or accurate explanations of how the final answer was arrived at—even if the answer is correct, as an example provided by Hendrycks et al. clearly demonstrates ([19] sec 4.1).”
PAL: Program-aided Language Models (2022, background): “…decompose natural language problems into steps and perform simple arithmetic operations, their performance falls dramatically when dealing with complex arithmetic (Hendrycks et al., 2021; Madaan & Yazdanbakhsh, 2022) or large numbers (Geva et al., 2020; Nogueira et al., 2021; Qian et al., 2022).”
TALM: Tool Augmented Language Models (2022, positive): “According to our solver tool, approximately 70% of the formulas in MathQA produce results that match their corresponding answers, similar to the ﬁndings in [Hendrycks et al., 2021”
From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline (2024, background): “Common ones include MMLU [16], MATH [17], GSM-8K [10], HumanEval [7], DROP [12], BigBench [37], Hel-laSwag [49], AGIEval [52], GPQA [35], as well as comprehensive collection like HELM [26].”
MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (2024, neutral): “The recent published GPT-4o has achieved remarkable performance boost (10+% improvement) on MATH [19], Chatbot Arena [11].”
LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, methodology): “For AtCoder and CodeForces problems, we use the standard input format (similar to Hendrycks et al. (2021)).”
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (2024, methodology): “…contamination, we follow Guo et al. (2024) to filter out web pages containing questions or answers from English mathematical benchmarks such as GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021) and Chinese benchmarks such as CMATH (Wei et al., 2023) and AGIEval (Zhong et al., 2023).”
C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models (2023, methodology): “For subjects with complex mathematical notations such as many subjects in the STEM category, we manually convert them into standard L A T E X formats, similar to Hendrycks et al. (2021b); Taylor et al. (2022).”