Successive Prompting for Decomposing Complex Questions

By Dheeru Dua, Shivanshu Gupta, Sameer Singh, Matt Gardner. 2022

153 citations recorded by the source corpus.

Answering complex questions that require making latent decisions is a challenging task, especially when limited supervision is available. Recent works leverage the capabilities of large language models (LMs) to perform complex question answering in a few-shot setting by demonstrating how to output intermediate rationalizations while solving the complex question in a single pass. We introduce “Successive Prompting” where, we iteratively break down a complex task into a simple task, solve it, and then repeat the process until we get the final solution. Successive prompting decouples the supervision for decomposing complex questions from the supervision for answering simple questions, allowing us to (1) have multiple opportunities to query in-context examples at each reasoning step (2) learn question decomposition separately from question answering, including using synthetic data, and (3) use bespoke (fine-tuned) components for reasoning steps where a large LM does not perform well. The intermediate supervision is typically manually written, which can be expensive to collect. We introduce a way to generate synthetic dataset which can be used to bootstrap model’s ability to decompose and answer intermediate questions. Our best model (with successive prompting) achieves an improvement in F1 of ~5% when compared with a state-of-the-art model with synthetic augmentations and few-shot version of the DROP dataset.

Subfields: Artificial Intelligence, Natural Language Processing, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Prompting & Reasoning

Citations in the corpus

Augmented Language Models: a Survey (2023, background): “(2022), many recent works employ in-context learning to do so (Yang et al., 2022a; Khot et al., 2022; Dua et al., 2022; Kazemi et al., 2022).”
Decomposed Prompting: A Modular Approach for Solving Complex Tasks (2022, background): “, 2023) and successive prompting (Dua et al., 2022) where one prompt/model is used to generate the subquestions needed to answer a complex question and a second prompt/model sequentially answers these sub-questions.”
Parsel: A Unified Natural Language Framework for Algorithmic Reasoning (2022, background): “…et al., 2020; Nye et al., 2021; Wei et al., 2022; Marasovi´c et al., 2021; Lampinen et al., 2022) and correspondingly, that this performance can be improved with further guidance and tool use (Zhou et al., 2022a; Zelikman et al., 2022; Yao et al., 2022; Uesato et al., 2022; Dua et al., 2022).”
LAMBADA: Backward Chaining for Automated Reasoning in Natural Language (2022, unknown): “Modular Reasoning: These approaches break the problem into smaller modules and use separate LMs to solve each module (Zhou et al., 2022; Khot et al., 2023; Sprague et al., 2022; Zhou et al., 2023; Dua et al., 2022; Wang et al., 2022; Schlag et al., 2023).”
Measuring Faithfulness in Chain-of-Thought Reasoning (2023, background): “Approaches include generating subquestions (Dua et al., 2022; Zhou et al., 2023), producing a tree of thoughts (Yao et al., 2023a), devising and executing a plan for answering the question (Wang et al., 2023), and having language models debate to reach an answer (Du et al., 2023), among others.”
Reasoning with Language Model Prompting: A Survey (2022, unknown): “…al., 2023), MAIEUTIC (Jung et al., 2022), Faithful (Creswell and Shanahan, 2022), Decomposed (Khot et al., 2023), Self-Ask (Press et al., 2022), Successive (Dua et al., 2022), LMLP (Zhang et al., 2022), LAMBADA (Kazemi et al., 2022), Iter-Decomp (Reppert et al., 2023) Process Optimization (§3.1.2)”
MoreHopQA: More Than Multi-hop Reasoning (2024, background): “Additionally, recent works (Dua et al., 2022; Khot et al., 2023; Press et al., 2023; Zhou et al., 2023) attempted to incorporate a question decomposition step into their prompts to improve model performance.”
Towards Reasoning in Large Language Models: A Survey (2022, background): “Furthermore, Dua et al. (2022) develop successive prompting, which iteratively decomposes a complex problem into a simple problem, with the next subproblem prediction having access to the”