MathPrompter: Mathematical Reasoning using Large Language Models

By Shima Imani, Liang Du, Harsh Shrivastava. Annual Meeting of the Association for Computational Linguistics, 2023

301 citations recorded by the source corpus.

Large Language Models (LLMs) have limited performance when solving arithmetic reasoning tasks and often provide incorrect answers. Unlike natural language understanding, math problems typically have a single correct answer, making the task of generating accurate solutions more challenging for LLMs. To the best of our knowledge, we are not aware of any LLMs that indicate their level of confidence in their responses which fuels a trust deficit in these models impeding their adoption. To address this deficiency, we propose ‘MathPrompter’, a technique that improves performance of LLMs on arithmetic problems along with increased reliance in the predictions. MathPrompter uses the Zero-shot chain-of-thought prompting technique to generate multiple algebraic expressions or python functions to solve the same math problem in different ways and thereby raise the confidence level in the output results. This is in contrast to other prompt based CoT methods, where there is no check on the validity of the intermediate steps followed. Our technique improves over state-of-the-art on the ‘MultiArith’ dataset (78.7% - 92.5%) evaluated using 175B parameter GPT-based LLM.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/b626560f19f815808a289ef5c24a17c57320da70

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

JudgeLM: Fine-tuned Large Language Models are Scalable Judges (2023, unknown): “However we think it is a common problem of large language models (Imani et al., 2023), and future advancement on base models (Yu et al., 2023a; Luo et al., 2023) would alleviate this problem.”
Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models (2023, methodology): “Most existing approaches are either limited to a small number of tools [33, 6, 47, 15, 37, 43] or rely on domain-specific tools [34, 52, 11, 51, 45], and thus are not easy to generalize to diverse queries.”
CodeMind: A Framework to Challenge Large Language Models for Code Reasoning (2024, background): “…tasks of different modalieties (Deshpande et al., 2021; Wu et al., 2023; Miceli-Barone et al., 2023; Bubeck et al., 2023; Wang et al., 2023; Imani et al., 2023; Luo et al., 2023; Huang et al., 2023; Valmeekam et al., 2022; Min et al., 2023), including natural language, visual data, math,…”
ToolQA: A Dataset for LLM Question Answering with External Tools (2023, methodology): “On the other hand, Program-of-Thought [6], PAL [13], MathPrompt [15], and Code4Struct [63] apply code-based tools to enhance LLMs’ abilities in question answering with a focus on tabular and math-related tasks.”
A Comprehensive Evaluation of Tool-Assisted Generation Strategies (2023, background): “, 2022) and complex mathematical reasoning challenges (Mishra et al., 2022; Imani et al., 2023).”
CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society (2023, methodology): “…instruction or prompting methods , such as Chain-of-Thought (CoT) [123], zero-shot-CoT [61], and ReAct [126], have recently been developed to enhance the performance of LLMs on reasoning, arithmetic and decision making tasks [134, 118, 52, 73, 31, 103, 43, 64, 132, 46, 133, 105, 128, 25, 81, 109].”
Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning (2023, background): “For mathematical reasoning, several works have shown that the integration of a calculator (Cobbe et al., 2021; Imani et al., 2023) or a Python interpreter (Gao et al.”
MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models (2023, background): “…the rapid development of large language models (LLMs) which emerge as the favored approach for various applications and demonstrate multi-dimensional abilities, including instruction following [7, 38, 55, 67], coding assistance [8, 36, 44, 51], and mathematical problem-solving [14, 28, 43, 79].”
Discrete Flow Matching (2024, background): “…as, code (Roziere et al., 2023; Li et al., 2023; Chen et al., 2021), biology (Zhang et al., 2024; Ferruz and Höcker, 2022; Madani et al., 2023), math (Romera-Paredes et al., 2024; Imani et al., 2023; Ahn et al., 2024), audio (Kreuk et al., 2022; Copet et al., 2024; Hassid et al., 2024) and more.”