Is a Question Decomposition Unit All We Need?

By Pruthvi H. Patel, Swaroop Mishra, Mihir Parmar, Chitta Baral. 2022

65 citations recorded by the source corpus.

Large Language Models (LMs) have achieved state-of-the-art performance on many Natural Language Processing (NLP) benchmarks. With the growing number of new benchmarks, we build bigger and more complex LMs. However, building new LMs may not be an ideal option owing to the cost, time and environmental impact associated with it. We explore an alternative route: can we modify data by expressing it in terms of the model’s strengths, so that a question becomes easier for models to answer? We investigate if humans can decompose a hard question into a set of simpler questions that are relatively easier for models to solve. We analyze a range of datasets involving various forms of reasoning and find that it is indeed possible to significantly improve model performance (24% for GPT3 and 29% for RoBERTa-SQuAD along with a symbolic calculator) via decomposition. Our approach provides a viable option to involve people in NLP research in a meaningful way. Our findings indicate that Human-in-the-loop Question Decomposition (HQD) can potentially provide an alternate path to building large LMs.

Subfields: Artificial Intelligence, Prompting & Reasoning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Prompting & Reasoning

In Question Answering & Reading Comprehension

Citations in the corpus

Ask Me Anything: A simple strategy for prompting language models (2022, methodology): “Others require users to manually rewrite task-inputs to the prescribed formats on a example-by-example basis in a task-specific manner [Mishra et al., 2021, Patel et al., 2022, Wu et al., 2022].”
MoreHopQA: More Than Multi-hop Reasoning (2024, background): “Patel et al. (2022) showed that human decomposition improves performance on complex questions.”
ToolQA: A Dataset for LLM Question Answering with External Tools (2023, background): “In terms of planning, current methods either enable LLMs to autonomously break down complex tasks into intermediate reasoning steps [45, 24, 47, 50, 65, 75, 31, 76], or encourage LLMs to self-reflect the previous decisions with environmental feedback [39, 76, 35, 8].”
How Many Data Samples is an Additional Instruction Worth? (2022, methodology): “In addition, many approaches have been proposed to improve model performance using instructions (Wu et al., 2022; Lin et al., 2021; Wang et al., 2022b; Luo et al., 2022; Kuznia et al., 2022; Patel et al., 2022; Mishra and Nouri, 2022).”
HELP ME THINK: A Simple Prompting Strategy for Non-experts to Create Customized Content with Models (2022, background): “…of thought (Wei et al., 2022) or scratchpad (Nye et al., 2021), majority voting (Wang et al., 2022b), reframing (Mishra et al., 2022a), least-to-most prompting (Zhou et al., 2022), question decomposition (Khot et al., 2020; Patel et al., 2022) have been shown to be effective across various tasks.”