Large Language Models are Zero-Shot Reasoners
By Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, Yusuke Iwasawa. 2022
7,591 citations recorded by the source corpus.
Pretrained large language models (LLMs) are widely used in many sub-fields of natural language processing (NLP) and generally known as excellent few-shot learners with task-specific exemplars. Notably, chain of thought (CoT) prompting, a recent technique for eliciting complex multi-step reasoning through step-by-step answer examples, achieved the state-of-the-art performances in arithmetics and symbolic reasoning, difficult system-2 tasks that do not follow the standard scaling laws for LLMs. While these successes are often attributed to LLMs' ability for few-shot learning, we show that LLMs are decent zero-shot reasoners by simply adding"Let's think step by step"before each answer. Experimental results demonstrate that our Zero-shot-CoT, using the same single prompt template, significantly outperforms zero-shot LLM performances on diverse benchmark reasoning tasks including arithmetics (MultiArith, GSM8K, AQUA-RAT, SVAMP), symbolic reasoning (Last Letter, Coin Flip), and other logical reasoning tasks (Date Understanding, Tracking Shuffled Objects), without any hand-crafted few-shot examples, e.g. increasing the accuracy on MultiArith from 17.7% to 78.7% and GSM8K from 10.4% to 40.7% with large InstructGPT model (text-davinci-002), as well as similar magnitudes of improvements with another off-the-shelf large model, 540B parameter PaLM. The versatility of this single prompt across very diverse reasoning tasks hints at untapped and understudied fundamental zero-shot capabilities of LLMs, suggesting high-level, multi-task broad cognitive capabilities may be extracted by simple prompting. We hope our work not only serves as the minimal strongest zero-shot baseline for the challenging reasoning benchmarks, but also highlights the importance of carefully exploring and analyzing the enormous zero-shot knowledge hidden inside LLMs before crafting finetuning datasets or few-shot exemplars.
Subfields: Artificial Intelligence, Information Systems, Safety Research, Large Language Models, Prompting & Reasoning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Luke S. Zettlemoyer
- Quoc V. Le
- Yoshua Bengio
- Sergey Levine
- Yejin Choi
In Information Systems
- Percy Liang
- Jimmy J. Lin
- Jianfeng Gao
- Tushar Khot
- Peter Clark
- Wayne Xin Zhao
- Ashish Sabharwal
- Ji-rong Wen
In Safety Research
- Denny Zhou
- Jason Wei
- Alessandro Stolfo
- Mrinmaya Sachan
- Kumar Shridhar
- Jie Huang
- Wenhu Chen
- Quoc V. Le
In Large Language Models
- Luke S. Zettlemoyer
- Yejin Choi
- Noah A. Smith
- Jianfeng Gao
- Hinrich Schütze
- Mohit Bansal
- Samuel R. Bowman
- Yi Tay
In Prompting & Reasoning
- Denny Zhou
- Luke S. Zettlemoyer
- Jie Huang
- Kevin Chen-Chuan Chang
- Swaroop Mishra
- Wenhu Chen
- Tushar Khot
- Peter Clark
Citations in the corpus
- Constitutional AI: Harmlessness from AI Feedback (2022, methodology): “Specifically, we ask language models to ‘think step-by-step’ [Kojima et al., 2022] and write out an argument explaining why one AI assistant response would be more harmless than another, before actually choosing the less harmful response.”
- ReAct: Synergizing Reasoning and Acting in Language Models (2022, neutral): “, 2022), zero-shotCoT (Kojima et al., 2022), and reasoning with self-consistency (Wang et al.”
- Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, neutral): “Prompt with correct chain-of-thought 17.1 Prompt with imperfect chain-of-thought 14.9 + Self-consistency (40 paths) 23.4 Prompt with equations 5.0 + Self-consistency (40 paths) 6.5 PaLM-540B Zero-shot CoT (Kojima et al., 2022) 43.0 + Self-consistency (40 paths) 69.2 Table 8: Self-consistency works u”
- Reasoning Like Program Executors (2022, background): “, GPT-3) can perform well on reasoning tasks (Wei et al., 2022; Kojima et al., 2022; Li et al., 2022b).”
- OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization (2022, background): “Each dataset has a single prompt that uses an instruction, that explicitly asks the model to generate a reasoning chain (Kojima et al., 2022), followed by examples in the few-shot setting that illustrate how the reasoning chain should be produced before the answer.”
- ChatGPT outperforms crowd workers for text-annotation tasks (2023, background): “…to recommend or even automate the labeling (Desmond et al., 2021); (v) using chain of thought prompting and other strategies to increase the performance of zero-shot reasoning (Kojima et al., 2022); (vi) and of course implementation of annotation tasks with GPT-4, as soon as availability permits.”
- The Capacity for Moral Self-Correction in Large Language Models (2023, methodology): “Finally, we explore a variant of Chain of Thought (CoT) prompting [28] in which we instruct the dialogue model to produce (and consider) text describing how it might follow the instructions before answering the question (Q+IF+CoT).”
- Galactica: A Large Language Model for Science (2022, background): “The capability set increases with scale, and recent work has highlighted reasoning capabilities at larger scales with a suitable prompting strategy (Wei et al., 2022; Chowdhery et al., 2022; Kojima et al., 2022; Lewkowycz et al., 2022).”
- Large Language Models Are Human-Level Prompt Engineers (2022, background): “One of the most influential recent works of prompt engineering was the discovery (Kojima et al., 2022) that LLMs could be made to give chain-of-thoughts simply by prepending “Let’s think step by step.”
- Scaling Instruction-Finetuned Language Models (2022, background): “Although the negative zero-shot CoT result on PaLM may appear to contradict the findings from Kojima et al. (2022), a closer comparison reveals that they are not inconsistent.”
- Large Language Models Can Self-Improve (2022, neutral): “The most consistent answer, which is not necessarily a correct answer, is selected by majority voting, denoted as ˜ y i = arg max y ij =1 I ( y i j = y i k ) .”
- Faithful Reasoning Using Large Language Models (2022, background): “More-over, current work leverages reasoning traces to boost performance – rather than to aid explain-ability or build trust – allowing for hallucination of “knowledge” during reasoning (Kojima et al., 2022; Wei et al., 2022).”
- Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, background): “, prompted), they can even carry out some degree of deduction and respond to questions that appear to require reasoning and inference [8, 9, 10, 11, 12, 13].”
- Rationale-Augmented Ensembles in Language Models (2022, methodology): “The results for the PaLM-540B model are shown in Table 3, Table 4 and Table 6, and give a comparison to two baseline approaches: (1) standard few-shot prompting without rationales (Brown et al., 2020), and (2) rationale-based prompting, including few-shot chain-of-thought (CoT) prompting (Wei et al.”
- Inverse scaling can become U-shaped (2022, methodology): “The 0-shot CoT approach proposed by Kojima et al. (2022) is one method that does not require manual prompt construction, but as we show in the additional experiment in Appendix F, the effectiveness of this approach is limited for the inverse scaling tasks.”