Solving Quantitative Reasoning Problems with Language Models

By Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Venkatesh Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo and 4 more. 2022

1,845 citations recorded by the source corpus.

Language models have achieved remarkable performance on a wide range of tasks that require natural language understanding. Nevertheless, state-of-the-art models have generally struggled with tasks that require quantitative reasoning, such as solving mathematics, science, and engineering problems at the college level. To help close this gap, we introduce Minerva, a large language model pretrained on general natural language data and further trained on technical content. The model achieves state-of-the-art performance on technical benchmarks without the use of external tools. We also evaluate our model on over two hundred undergraduate-level problems in physics, biology, chemistry, economics, and other sciences that require quantitative reasoning, and find that the model can correctly answer nearly a third of them.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, methodology): “We also add a new subcorpora for arXiv and math webpages from the dataset used in Lewkowycz et al. (2022) in the proportion of 5%.”
AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models (2023, positive): “For MATH, we use the same instances as in the appendices of Lewkowycz et al. (2022).”
Augmented Language Models: a Survey (2023, methodology): “Using few-shot CoT prompting, Minerva (Lewkowycz et al., 2022) achieves excellent performance on math benchmarks such as GSM8K (Cobbe et al., 2021).”
Galactica: A Large Language Model for Science (2022, methodology): “Our 120B model achieves a score of 20.4% versus PaLM 540B’s 8.8% on MATH (Chowdhery et al., 2022; Lewkowycz et al., 2022).”
Scaling Instruction-Finetuned Language Models (2022, background): “The prompts for STEM datasets are from Lewkowycz et al. (2022), with minor formatting ﬁxes.”
Llemma: An Open Language Model For Mathematics (2023, methodology): “We follow Lewkowycz et al. (2022) and use their provided four-example chain-of-thought prompt.”
Large Language Models Can Self-Improve (2022, neutral): “…et al., 2022) crowd-sourced many human answers for diverse sets of text instructions to better align their model to human instructions; Minerva (Lewkowycz et al., 2022) parsed the full ArXiv database carefully for relevant articles to excel on challenging competitive math and science datasets.”
Improving alignment of dialogue agents via targeted human judgements (2022, background): “We believe these limitations are best addressed via multistep reasoning (Creswell et al., 2022; Dohan et al., 2022; Lewkowycz et al., 2022), with the further benefit of producing interpretable reasoning traces.”
PAL: Program-aided Language Models (2022, background): “In fact, even when fine-tuning a PaLM-based model on 164B tokens of explicit mathematical content, its two most common failures are reportedly “incorrect reasoning” and “incorrect calculation” (Lewkowycz et al., 2022).”
Learning by Distilling Context (2022, background): “Many recent works show that language models perform better when it is required to generate a chain of reasoning steps before outputting the ﬁnal answer (Zhou et al., 2021; Nye et al., 2021b; Cobbe et al., 2021; Wei et al., 2022b; Zhou et al., 2022; Lewkowycz et al., 2022).”
PaLM-E: An Embodied Multimodal Language Model (2023, background): “, 2022), math problem solving (Lewkowycz et al., 2022; Polu et al., 2022), and code writing (Chen et al.”
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (2024, methodology): “…the DeepSeekMath Corpus, a high-quality dataset of 120B tokens from web pages filtered for mathematical content, which is almost 7 times the size of the math web pages used by Minerva (Lewkowycz et al., 2022a) and 9 times the size of the recently released OpenWebMath (Paster et al., 2023).”
PaLM 2 Technical Report (2023, methodology): “Recently, Minerva (Lewkowycz et al., 2022) achieved significant gains on quantitative reasoning tasks by fine-tuning PaLM on scientific and mathematical content from the Web.”
Transcending Scaling Laws with 0.1% Extra Compute (2022, neutral): “…new research across multiple fronts, e.g., scaling laws [Kaplan et al., 2020, Hoffmann et al., 2022, Tay et al., 2022a], emergent abilities [Wei † Work completed while at Google. et al., 2022a, Ganguli et al., 2022], reasoning capabilities [Wei et al., 2022b, Lewkowycz et al., 2022], inter alia.”
Let's Verify Step by Step (2023, methodology): “Similar to Lewkowycz et al. (2022) we construct a large-scale dataset of high-quality math-relevant tokens for use in a lightweight pretraining stage, before finetuning on comparably smaller datasets like MATH and PRM800K.”
Programming Is Hard - Or at Least It Used to Be: Educational Opportunities and Challenges of AI Code Generation (2022, methodology): “for coding decisions, similar to those generated by Minerva for mathematics problems [22].”