Chain of Thought Prompting Elicits Reasoning in Large Language Models
By Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Fei Xia, Quoc Le, Quoc V. Le, Denny Zhou. Neural Information Processing Systems, 2022
20,310 citations recorded by the source corpus.
We explore how generating a chain of thought -- a series of intermediate reasoning steps -- significantly improves the ability of large language models to perform complex reasoning. In particular, we show how such reasoning abilities emerge naturally in sufficiently large language models via a simple method called chain of thought prompting, where a few chain of thought demonstrations are provided as exemplars in prompting. Experiments on three large language models show that chain of thought prompting improves performance on a range of arithmetic, commonsense, and symbolic reasoning tasks. The empirical gains can be striking. For instance, prompting a 540B-parameter language model with just eight chain of thought exemplars achieves state of the art accuracy on the GSM8K benchmark of math word problems, surpassing even finetuned GPT-3 with a verifier.
Subfields: Artificial Intelligence, Computer Networks and Communications, Computer Vision and Pattern Recognition, Large Language Models, Prompting & Reasoning
View Paper: https://arxiv.org/abs/2201.11903
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Quoc V. Le
- Yoshua Bengio
- Yejin Choi
- Graham Neubig
In Computer Networks and Communications
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Ming-Hsuan Yang
- Andrew Zisserman
- Cordelia Schmid
In Large Language Models
- Luke S. Zettlemoyer
- Yejin Choi
- Hinrich Schütze
- Mohit Bansal
- Timo Schick
- Jianfeng Gao
- Noah A. Smith
- Furu Wei
In Prompting & Reasoning
- Luke S. Zettlemoyer
- Denny Zhou
- Hannaneh Hajishirzi
- Yejin Choi
- Jie Huang
- Swaroop Mishra
- Mohit Bansal
- Kevin Chen-Chuan Chang
Citations in the corpus
- PaLM: Scaling Language Modeling with Pathways (2022, methodology): “In Section 6.3, we empirically demonstrated how chain-of-thought prompting (Wei et al., 2022b) can drastically improve prediction accuracy in multi-step reasoning tasks.”
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, background): “Earlier work (Brown et al. (2020), Wei et al. (2022a), Du et al. (2022) in measuring such dataset contamination considered an example from an evaluation set to be “contaminated” if there existed a collision between a high-order n-gram (generally, n = 13) from the sample and the training data.”
- Constitutional AI: Harmlessness from AI Feedback (2022, methodology): “Chain-of-Thought Prompting We also experimented with using Chain-of-Thought (CoT) prompting [Wei et al., 2022] on the feedback model to generate labels.”
- ReAct: Synergizing Reasoning and Acting in Language Models (2022, methodology): “answers from questions in arithmetic, commonsense, and symbolic reasoning tasks (Wei et al., 2022).”
- Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, methodology): “In an effort to address this shortcoming, Wei et al. (2022) have proposed chain-of-thought prompting , where a language model is prompted to generate a series of short sentences that mimic the reasoning process a person might employ in solving a task.”
- Reasoning Like Program Executors (2022, background): “Reasoning in Giant Language Models Recent works demonstrate that with proper prompting (e.g., chain-of-thoughts prompting), giant language models (e.g., GPT-3) can perform well on reasoning tasks (Wei et al., 2022; Kojima et al., 2022; Li et al., 2022b).”
- Finetuned Language Models Are Zero-Shot Learners (2021, background): “, 2021; Qin & Eisner, 2021), as well as work that prompts models to produce specialized outputs (Wei et al., 2022).”
- OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization (2022, methodology): “…across three levels of generalization: 1) model performance on tasks from fully held-out task categories not used for tuning, as in prior work (Wei et al., 2022a; Sanh et al., 2022), and additionally, 2) performance on unseen tasks from categories seen during instructiontuning, and, 3)…”
- Emergent Abilities of Large Language Models (2022, neutral): “Although Wei et al. (2022a) initially found that instruction-based finetuning only worked for 68B parameter or larger decoder-only models, Sanh et al. (2022) induced similar behavior in a 11B model with an encoder-decoder architecture, which typically has higher performance after finetuning than…”
- CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis (2022, background): “Prompting pre-trained language models with intermediate information to improve task performance has attracted interest (Nye et al., 2021; Wei et al., 2022).”
- AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models (2023, positive): “Chain-of-Thought (CoT) prompting (Wei et al., 2022) enables large language models to break down a complex question to a series of decomposed reasoning steps.”
- Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor (2022, background): “FLAN-T5 Chung et al. (2022) fine-tune T5 on a collection of tasks phrased as instructions in multiple prompting setups (zero-shot, few-shot, Chainof-Thought (Wei et al., 2022)), achieving impressive zero-shot generalization capabilities.”
- Galactica: A Large Language Model for Science (2022, background): “We believe this adds another reasoning method to the deep learning toolkit, alongside the existing chain-of-thought approach that has been well explored recently (Wei et al., 2022; Suzgun et al., 2022).”
- Large Language Models Are Human-Level Prompt Engineers (2022, background): “Early works (Nye et al., 2021; Betz et al., 2021; Wei et al., 2022b) on chain-of-thought used fine-tuning or in-context learning to get LLMs to show their work for such problems.”