Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks

By Wenhu Chen, Xueguang Ma, Xinyi Wang, William W. Cohen. 2022

1,410 citations recorded by the source corpus.

Recently, there has been significant progress in teaching language models to perform step-by-step reasoning to solve complex numerical reasoning tasks. Chain-of-thoughts prompting (CoT) is by far the state-of-art method for these tasks. CoT uses language models to perform both reasoning and computation in the multi-step `thought' process. To disentangle computation from reasoning, we propose `Program of Thoughts' (PoT), which uses language models (mainly Codex) to express the reasoning process as a program. The computation is relegated to an external computer, which executes the generated programs to derive the answer. We evaluate PoT on five math word problem datasets (GSM, AQuA, SVAMP, TabMWP, MultiArith) and three financial-QA datasets (FinQA, ConvFinQA, TATQA) for both few-shot and zero-shot setups. Under both few-shot and zero-shot settings, PoT can show an average performance gain over CoT by around 12\% across all the evaluated datasets. By combining PoT with self-consistency decoding, we can achieve SoTA performance on all math problem datasets and near-SoTA performance on financial datasets. All of our data and code are released in Github https://github.com/wenhuchen/Program-of-Thoughts

Subfields: Artificial Intelligence, Computer Science Applications, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Percy Liang
  4. Luke S. Zettlemoyer
  5. Yejin Choi
  6. Graham Neubig
  7. Caiming Xiong
  8. Quoc V. Le

In Computer Science Applications

  1. Denny Zhou
  2. Kevin Chen-Chuan Chang
  3. Jie Huang
  4. Luke S. Zettlemoyer
  5. Ramakanth Pasunuru
  6. Thomas Scialom
  7. Timo Schick
  8. Pan Lu

In Prompting & Reasoning

  1. Denny Zhou
  2. Luke S. Zettlemoyer
  3. Jie Huang
  4. Wenhu Chen
  5. Xinyun Chen
  6. Kevin Chen-Chuan Chang
  7. Hannaneh Hajishirzi
  8. Tengyu Ma

Citations in the corpus