Least-to-Most Prompting Enables Complex Reasoning in Large Language Models
By Denny Zhou, Nathanael Scharli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Dale Schuurmans, Olivier Bousquet, Quoc V. Le and 1 more. 2022
1,834 citations recorded by the source corpus.
Chain-of-thought prompting has demonstrated remarkable performance on various natural language reasoning tasks. However, it tends to perform poorly on tasks which requires solving problems harder than the exemplars shown in the prompts. To overcome this challenge of easy-to-hard generalization, we propose a novel prompting strategy, least-to-most prompting. The key idea in this strategy is to break down a complex problem into a series of simpler subproblems and then solve them in sequence. Solving each subproblem is facilitated by the answers to previously solved subproblems. Our experimental results on tasks related to symbolic manipulation, compositional generalization, and math reasoning reveal that least-to-most prompting is capable of generalizing to more difficult problems than those seen in the prompts. A notable finding is that when the GPT-3 code-davinci-002 model is used with least-to-most prompting, it can solve the compositional generalization benchmark SCAN in any split (including length split) with an accuracy of at least 99% using just 14 exemplars, compared to only 16% accuracy with chain-of-thought prompting. This is particularly noteworthy because neural-symbolic models in the literature that specialize in solving SCAN are trained on the entire training set containing over 15,000 examples. We have included prompts for all the tasks in the Appendix.
Subfields: Artificial Intelligence, Large Language Models, Prompting & Reasoning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Percy Liang
- Jianfeng Gao
- Quoc V. Le
- Graham Neubig
- Yoshua Bengio
- Yejin Choi
In Large Language Models
- Luke S. Zettlemoyer
- Hinrich Schütze
- Timo Schick
- Noah A. Smith
- Yi Tay
- Graham Neubig
- Quoc V. Le
- Michael Lewis
In Prompting & Reasoning
- Denny Zhou
- Jie Huang
- Kevin Chen-Chuan Chang
- Luke S. Zettlemoyer
- Wenhu Chen
- Tushar Khot
- Ashish Sabharwal
- Swaroop Mishra
Citations in the corpus
- ReAct: Synergizing Reasoning and Acting in Language Models (2022, neutral): “Several follow-up works have since been performed, including least-to-most prompting for solving complicated tasks (Zhou et al., 2022), zero-shotCoT (Kojima et al.”
- Emergent Abilities of Large Language Models (2022, positive): “A recent prompting strategy called chain-of-thought prompting enables language models to solve such problems by guiding them to produce a sequence of intermediate steps before giving the final answer (Cobbe et al., 2021; Wei et al., 2022b; Zhou et al., 2022).”
- Augmented Language Models: a Survey (2023, background): “, 2022) combined with compositionality (Zhou et al., 2022) capabilities made LLMs able to execute various tasks such as language understanding or conditional and 1See e.”
- Large Language Models Can Self-Improve (2022, neutral): “Zhou et al. (2022) decompose the questions into multiple sub-questions, and ask the language model to solve each sub-question sequentially.”
- Faithful Reasoning Using Large Language Models (2022, neutral): “Similar to our own model Zhou et al. (2022) go one-step further and split each reasoning step in two: first asking an intermediate question and second, answering that question.”
- PAL: Program-aided Language Models (2022, methodology): “This approach of prompting the model to first generate a reasoning process ttest improves the accuracy of the answer ytest across a wide range of tasks (Wang et al., 2022b; Wei et al., 2022; Zhou et al., 2022; Wang et al., 2022c).”
- Learning by Distilling Context (2022, background): “Many recent works show that language models perform better when it is required to generate a chain of reasoning steps before outputting the final answer (Zhou et al., 2021; Nye et al., 2021b; Cobbe et al., 2021; Wei et al., 2022b; Zhou et al., 2022; Lewkowycz et al., 2022).”
- Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models (2023, neutral): “For the few-shot setting, the LLMs perform CoT reasoning with several demonstrations [58, 41], and it turns out that the LLMs can acquire better abilities to solve complex problems.”
- Multimodal Chain-of-Thought Reasoning in Language Models (2023, positive): “However, existing studies related to CoT reasoning are largely isolated in the language modality (Wang et al., 2022b; Zhou et al., 2022; Lu et al., 2022b; Fu et al., 2022), with little consideration of multimodal scenarios.”
- C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models (2023, background): “Compared to zero-shot COT, the few-shot version is more commonly used and achieves the state-of-the-art performance on various tasks (Gao et al., 2022; Wang et al., 2023; Zhou et al., 2023; Xie et al., 2023).”
- Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks (2022, background): “Later on, several other works (Drozdov et al., 2022; Zhou et al., 2022; Nye et al., 2021) also propose different approaches to utilize LLMs to solve reasoning tasks by allowing intermediate steps.”
- Transcending Scaling Laws with 0.1% Extra Compute (2022, neutral): “, reasoning (Wei et al., 2022b; Wang et al., 2022; Zhou et al., 2022; Drozdov et al., 2022), math (Lewkowycz et al.”
- Measuring and Narrowing the Compositionality Gap in Language Models (2022, background): “…problems has been previously explored in both language (Graves, 2016; Seo et al., 2017; Dong et al., 2019; Schwartz et al., 2020; Schwarzschild et al., 2021; Nye et al., 2021; Wei et al., 2022b; Zhou et al., 2022) and vision (Bolukbasi et al., 2017; Huang et al., 2018; Wang et al., 2018, 2021).”
- Decomposed Prompting: A Modular Approach for Solving Complex Tasks (2022, background): “Least-to-most prompting (Zhou et al., 2022) also proposes a similar solution but differs in two key aspects (a) it has to identify all the subproblems in one-shot instead of our iterative top-down decomposition (b) it has to learn to identify the relevant answers from the previous solutions which we”
- Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training (2022, background): “It dates back to at least Andreas et al. (2018) and saw renewed interest in the past few months due to the prevalence of large PLMs. Andreas et al. (2018) and Vong and Lake (2022) learn natural language descriptions that function as few-shot classifiers within an image-text matching model.”
- DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction (2023, methodology): “We manually examined these failures and classified them into six categories as shown in Figure 1 and discussed next.”
- Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model (2022, background): “” In chain-of-thought and least-to-most prompting, the model is prompted to produce an explanation alongside its answer, with questions focusing on arithmetic and commonsense reasoning (Kojima et al., 2022; Wei et al., 2022; Zhou et al., 2022).”
- DreamLLM: Synergistic Multimodal Comprehension and Creation (2023, background): “The sensitivity of LLMs to human prompts is a known issue (Wei et al., 2022b; Wang et al., 2023b; Zhou et al., 2023), a challenge that extends to MLLMs.”