Show Your Work: Scratchpads for Intermediate Computation with Language Models

By Maxwell Nye, Anders Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, D. Luan and 2 more. 2021

1,075 citations recorded by the source corpus.

Large pre-trained language models perform remarkably well on tasks that can be done"in one pass", such as generating realistic text or synthesizing computer programs. However, they struggle with tasks that require unbounded multi-step computation, such as adding integers or executing programs. Surprisingly, we find that these same models are able to perform complex multi-step computations -- even in the few-shot regime -- when asked to perform the operation"step by step", showing the results of intermediate computations. In particular, we train transformers to perform multi-step computations by asking them to emit intermediate computation steps into a"scratchpad". On a series of increasingly complex tasks ranging from long addition to the execution of arbitrary programs, we show that scratchpads dramatically improve the ability of language models to perform multi-step computations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Luke S. Zettlemoyer
  3. Percy Liang
  4. Jianfeng Gao
  5. Quoc V. Le
  6. Yejin Choi
  7. Graham Neubig
  8. Yoshua Bengio

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Xiaogang Wang
  5. Jitendra Malik
  6. Ming-Hsuan Yang
  7. Andrew Zisserman
  8. Bernt Schiele

In Natural Language Processing

  1. Mohit Bansal
  2. Luke S. Zettlemoyer
  3. Yejin Choi
  4. Graham Neubig
  5. Hannaneh Hajishirzi
  6. Percy Liang
  7. Noah A. Smith
  8. Jianfeng Gao

In Prompting & Reasoning

  1. Luke S. Zettlemoyer
  2. Denny Zhou
  3. Hannaneh Hajishirzi
  4. Yejin Choi
  5. Wenhu Chen
  6. Swaroop Mishra
  7. Jason Wei
  8. Tushar Khot

Citations in the corpus