Decomposed Prompting: A Modular Approach for Solving Complex Tasks

By Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, Ashish Sabharwal. 2022

735 citations recorded by the source corpus.

Few-shot prompting is a surprisingly powerful way to use Large Language Models (LLMs) to solve various tasks. However, this approach struggles as the task complexity increases or when the individual reasoning steps of the task themselves are hard to learn, especially when embedded in more complex tasks. To address this, we propose Decomposed Prompting, a new approach to solve complex tasks by decomposing them (via prompting) into simpler sub-tasks that can be delegated to a library of prompting-based LLMs dedicated to these sub-tasks. This modular structure allows each prompt to be optimized for its specific sub-task, further decomposed if necessary, and even easily replaced with more effective prompts, trained models, or symbolic functions if desired. We show that the flexibility and modularity of Decomposed Prompting allows it to outperform prior work on few-shot prompting using GPT3. On symbolic reasoning tasks, we can further decompose sub-tasks that are hard for LLMs into even simpler solvable sub-tasks. When the complexity comes from the input length, we can recursively decompose the task into the same task but with smaller inputs. We also evaluate our approach on textual multi-step reasoning tasks: on long-context multi-hop QA task, we can more effectively teach the sub-tasks via our separate sub-tasks prompts; and on open-domain multi-hop QA, we can incorporate a symbolic information retrieval within our decomposition framework, leading to improved performance on both tasks. Datasets, Code and Prompts available at https://github.com/allenai/DecomP.

Subfields: Artificial Intelligence, Large Language Models, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Prompting & Reasoning

Citations in the corpus

Augmented Language Models: a Survey (2023, background): “(2022), many recent works employ in-context learning to do so (Yang et al., 2022a; Khot et al., 2022; Dua et al., 2022; Kazemi et al., 2022).”
Multimodal Chain-of-Thought Reasoning in Language Models (2023, neutral): “Similarly, Khot et al. (2022) used diverse decomposition structures and designed different prompts to answer each sub-question.”
Visual Programming: Compositional visual reasoning without training (2022, methodology): “While CoT relies on the ability of LLMs to both generate a reasoning path and execute it, approaches similar to VISPROG have been applied to language tasks, where a decomposer pompt [15] is used first to generate a sequence of sub-tasks which are then handled by sub-task handlers.”
Measuring and Narrowing the Compositionality Gap in Language Models (2022, methodology): “In parallel to our work Khot et al. (2023) and Yao et al. (2023) present methods similar to self-ask, but they do not present our findings on the compo-sitionality gap, integrate their prompt with a web search engine or present new datasets as in our Compositional Celebrities and Bamboogle.”
Is ChatGPT a General-Purpose Natural Language Processing Task Solver? (2023, background): “Most recent studies mainly focused on how to improve manual-CoT, including optimizing the demonstration selection (Rubin et al., 2022; Fu et al., 2022; Lu et al., 2022b) and optimizing the quality of reasoning chains (Khot et al., 2022; Chen et al., 2022).”
Large Language Models Can Be Easily Distracted by Irrelevant Context (2023, background): “…al., 2022; Suzgun et al., 2022; Shi et al., 2022b, inter alia ), problem decomposition (Zhou et al., 2022; Drozdov et al., 2022; Dohan et al., 2022; Khot et al., 2022; Press et al., 2022, inter alia ), generating programs (Austin et al., 2021; Chowdhery et al., 2022; Gao et al., 2022; Chen et al.,…”
ToolQA: A Dataset for LLM Question Answering with External Tools (2023, background): “In terms of planning, current methods either enable LLMs to autonomously break down complex tasks into intermediate reasoning steps [45, 24, 47, 50, 65, 75, 31, 76], or encourage LLMs to self-reflect the previous decisions with environmental feedback [39, 76, 35, 8].”
ReFT: Reasoning with Reinforced Fine-Tuning (2024, background): “Most of them attempted to make CoT comprehensive and fine-grained to present the step-by-step reasoning solutions (Nye et al., 2021; Fu et al., 2023; Zhou et al., 2023b; Khot et al., 2023; Imani et al., 2023; Miao et al., 2023).”
Teaching Large Language Models to Self-Debug (2023, background): “For example, decomposed prompting (Khot et al., 2022) delegates subproblems to other LLM instances that are prompted to specialize in specific tasks.”
DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines (2023, methodology): “A growing body of work has revealed that, especially with instruction tuning (Ouyang et al., 2022), we can elicit sophisticated behavior via prompting (Wei et al., 2022; Wang et al., 2022b; Press et al., 2022; Yao et al., 2022; Khot et al., 2022; Madaan et al., 2023).”
A Comprehensive Evaluation of Tool-Assisted Generation Strategies (2023, methodology): “There are also strategies such as Decom-posed Prompting (Khot et al., 2023) that are generalizations of the previous strategies.”
Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models (2023, background): “Tool-Augmented Language Models In recent years, the development of large language models (LLMs) [46, 7, 8, 50, 4, 39, 40] has made tremendous progress, and further stimulate the research in prompting learning [53, 31, 21] and instruction learning [50, 60, 44, 10].”
A Survey of Deep Learning for Mathematical Reasoning (2022, neutral): “Similarly, Khot et al. (2022) leverage diverse decomposition structures and use different prompts to answer each sub-question.”
Training Large Language Models to Reason in a Continuous Latent Space (2024, positive): “This includes prompting LLMs (Wei et al., 2022; Khot et al., 2022; Zhou et al., 2022), or training LLMs to generate reasoning chains, either with supervised finetuning (Yue et al., 2023; Yu et al., 2023) or reinforcement learning (Wang et al., 2024; Havrilla et al., 2024; Shao et al., 2024; Yu et…”
Large Language Models as Analogical Reasoners (2023, background): “…with more structured algorithms and search methods (Khot et al., 2022; Drozdov et al., 2022; Zelikman et al., 2022; Yao et al., 2023; Press et al., 2022; Khattab et al., 2022; Jung et al., 2022), as well as longer-horizon action and planning (Yao et al., 2022; Hao et al., 2023; Park et al., 2023).”
Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity (2024, background): “Specifically, Khattab et al. (2022), Press et al. (2023), Pereira et al. (2023) and Khot et al. (2023) proposed to first decompose the multi-hop queries into simpler single-hop queries, repeatedly access the LLMs and retriever to solve these sub-queries, and merge their solutions to formulate a…”