Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks
By Wenhu Chen, Xueguang Ma, Xinyi Wang, William W. Cohen. 2022
1,410 citations recorded by the source corpus.
Recently, there has been significant progress in teaching language models to perform step-by-step reasoning to solve complex numerical reasoning tasks. Chain-of-thoughts prompting (CoT) is by far the state-of-art method for these tasks. CoT uses language models to perform both reasoning and computation in the multi-step `thought' process. To disentangle computation from reasoning, we propose `Program of Thoughts' (PoT), which uses language models (mainly Codex) to express the reasoning process as a program. The computation is relegated to an external computer, which executes the generated programs to derive the answer. We evaluate PoT on five math word problem datasets (GSM, AQuA, SVAMP, TabMWP, MultiArith) and three financial-QA datasets (FinQA, ConvFinQA, TATQA) for both few-shot and zero-shot setups. Under both few-shot and zero-shot settings, PoT can show an average performance gain over CoT by around 12\% across all the evaluated datasets. By combining PoT with self-consistency decoding, we can achieve SoTA performance on all math problem datasets and near-SoTA performance on financial datasets. All of our data and code are released in Github https://github.com/wenhuchen/Program-of-Thoughts
Subfields: Artificial Intelligence, Computer Science Applications, Prompting & Reasoning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Luke S. Zettlemoyer
- Yejin Choi
- Graham Neubig
- Caiming Xiong
- Quoc V. Le
In Computer Science Applications
- Denny Zhou
- Kevin Chen-Chuan Chang
- Jie Huang
- Luke S. Zettlemoyer
- Ramakanth Pasunuru
- Thomas Scialom
- Timo Schick
- Pan Lu
In Prompting & Reasoning
- Denny Zhou
- Luke S. Zettlemoyer
- Jie Huang
- Wenhu Chen
- Xinyun Chen
- Kevin Chen-Chuan Chang
- Hannaneh Hajishirzi
- Tengyu Ma
Citations in the corpus
- Augmented Language Models: a Survey (2023, methodology): “Similarly, Drori et al. (2022); Chen et al. (2022b) prompts Codex (Chen et al., 2021) to generate executable code-based solutions to university-level problems, math word problems, or financial QA.”
- ViperGPT: Visual Inference via Python Execution for Reasoning (2023, methodology): “In the natural language processing domain, they have been aimed at using exter-nal tools [46, 40], or for structured reasoning using Codex [34, 54, 14, 10].”
- Multimodal Chain-of-Thought Reasoning in Language Models (2023, neutral): “To elicit CoT reasoning in multimodality, we advocate a Multimodal-1”
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (2024, methodology): “We evaluate program-aided mathematical reasoning on GSM8K and MATH using few-shot program-of-thought prompting (Chen et al., 2022; Gao et al., 2023).”
- MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts (2023, methodology): “…and Claude-2 (Anthropic, 2023) in zero-shot and two-shot settings with Chain-of-Thought (CoT) (Wei et al., 2022b) and Program-of-Thought (PoT) (Chen et al., 2022b), (b) Augmented-LLMs where the LLMs are provided with additional visual information including the generated image captions from…”
- Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning (2023, background): “, 2023) or a Python interpreter (Gao et al., 2022; Chen et al., 2022) into LLMs can greatly enhance performance by offloading numerical calculations.”
- Chain of Thought Empowers Transformers to Solve Inherently Serial Problems (2024, background): “The success of CoT has even been extended to multi-modal QA (Zhang et al., 2023; Yang et al., 2023) and code generation (Chen et al., 2022; Gao et al., 2022).”
- Large Language Models as Analogical Reasoners (2023, methodology): “A common approach is prompting LLMs to generate intermediate reasoning steps, as demonstrated by the chain-of-thought method (Wei et al., 2022b; Kojima et al., 2022; Zhou et al., 2022; Wang et al., 2022), which assists LLMs in tackling complex reasoning tasks.”
- Improving large language models for clinical named entity recognition via prompt engineering (2024, positive): “39 Second, our few-shot learning approaches were relatively simple, and we plan to investigate other approaches such as the chain-of-thoughts method, 40–42 hoping to yield better results.”
- DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines (2023, background): “Unfortunately, LMs are known to be sensitive to how they are prompted for each task, and this is exacerbated in pipelines where multiple LM calls have to interact effectively.”
- Large Language Models as Tool Makers (2023, background): “However, their primary focus is on using Python executors to accurately solve sub-steps involving arithmetic reasoning, similar to Gao et al. (2023); Chen et al. (2022).”
- A Comprehensive Evaluation of Tool-Assisted Generation Strategies (2023, methodology): “Inline strategies such as Toolformer (Schick et al., 2023) 1 , ART (Paranjape et al., 2023), inter alia (Chen et al., 2022; Gao et al., 2023b; Lyu et al., 2023) demonstrate tool usage with pre-defined words or tokens and tool arguments, halt generation when those tokens and arguments are generated,…”
- Teaching Large Language Models to Self-Debug (2023, background): “Several prompting methods have been proposed to unlock latent abilities of large language models (Wei et al., 2022; Nye et al., 2021; Kojima et al., 2022; Zhou et al., 2023; Khot et al., 2022; Zhou et al., 2022; Gao et al., 2022; Chen et al., 2022).”
- ToolRL: Reward is All Tool Learning Needs (2025, positive): “…to enhance the capabilities of LLMs. Early studies introduced the concept of equipping LLMs with external tools to overcome their inherent limitations (Schick et al., 2023; Qin et al., 2024b; Yao et al., 2023), such as program executors (Chen et al., 2022) and search engines (Vu et al., 2023).”
- ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases (2023, background): “…of LLMs, prompting them to interact with various tools, ranging from highly specialized ones such as code interpreters (Gao et al. 2022; Chen et al. 2022), search engines (Yao et al. 2022), retrieval models (Khattab et al. 2023) and AI models (Shen et al. 2023; Lu et al. 2023), to more…”
- MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning (2023, methodology): “In contrast, prompts in the format of code like Program-of-Thought (PoT) approaches (Chen et al., 2022) and PAL (Madaan et al., 2022; Gao et al., 2023) utilize external tools (i.e., Python interpreter) to greatly simplify the math solving process.”
- A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications (2024, neutral): “Wang et al. [2022] introduced self-consistency, a decoding strategy enhancing reasoning performance compared to greedy decoding in CoT prompting.”