Unsupervised Question Decomposition for Question Answering

By Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, Douwe Kiela. 2020

206 citations recorded by the source corpus.

We aim to improve question answering (QA) by decomposing hard questions into simpler sub-questions that existing QA systems are capable of answering. Since labeling questions with decompositions is cumbersome, we take an unsupervised approach to produce sub-questions, also enabling us to leverage millions of questions from the internet. Specifically, we propose an algorithm for One-to-N Unsupervised Sequence transduction (ONUS) that learns to map one hard, multi-hop question to many simpler, single-hop sub-questions. We answer sub-questions with an off-the-shelf QA model and give the resulting answers to a recomposition model that combines them into a final answer. We show large QA improvements on HotpotQA over a strong baseline on the original, out-of-domain, and multi-hop dev sets. ONUS automatically learns to decompose different kinds of questions, while matching the utility of supervised and heuristic decomposition methods for QA and exceeding those methods in fluency. Qualitatively, we find that using sub-questions is promising for shedding light on why a QA system makes a prediction.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies (2021, background): “, 2020), and to train models that generate question decompositions (Perez et al., 2020; Khot et al., 2020b; Min et al., 2019).”
Augmented Language Models: a Survey (2023, methodology): “…employ problem decomposition can either then solve the sub-problems independently, where these answers are aggregated to generate the final answer (Perez et al., 2020; Min et al., 2019), or solve the sub-problems sequentially, where the solution to the next sub-problem depends on the answer to…”
Internet-augmented language models through few-shot prompting for open-domain question answering (2022, background): “, 2021), where complex queries could be decomposed into simpler sub-queries, akin to approaches in question decomposition (Min et al., 2019; Perez et al., 2020).”
Recursively Summarizing Books with Human Feedback (2021, background): “While Dayan and Hinton (1993) and Christiano et al. (2018) only tried this on simple algorithmic tasks, Perez et al. (2020) and Rajani et al. (2019) use similar ideas for question-answering tasks using a single step of decomposition.”
Least-to-Most Prompting Enables Complex Reasoning in Large Language Models (2022, background): “Perez et al. (2020) improves question answering (QA) by decomposing a multi-hop question to several single-hop subquestions.”
Discovering Language Model Behaviors with Model-Written Evaluations (2022, background): “, 2021), generating subquestions for a question (Perez et al., 2020), and question-answering and summarization (Joshi et al.”
True Few-Shot Learning with Language Models (2021, background): “Other approaches assume unlabeled data and develop unsupervised model selection techniques [74–76].”
Unsupervised Commonsense Question Answering with Self-Talk (2020, background): “…(Saeidi et al., 2018; Pan et al., 2019), simplifying questions to make them more easily answerable (Buck et al., 2018; Talmor and Berant, 2018; Perez et al., 2020), to us-ing questions as means for other purposes such as sentence representation and summarization (Guo et al., 2018; Potash and…”
Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models (2021, methodology): “Perez et al. (2020) also use a text-to-text model to generate sub-questions for HotpotQA.”
Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval (2020, background): “, 2018) and remains an active research area in NLP (Qi et al., 2019; Nie et al., 2019; Min et al., 2019; Zhao et al., 2020; Asai et al., 2020; Perez et al., 2020).”
Decomposed Prompting: A Modular Approach for Solving Complex Tasks (2022, methodology): “…for Multi-Step Reasoning Our work follows a long literature in NLP on neural modular modeling architectures (Andreas et al., 2016; Talmor & Berant, 2018; Jiang & Bansal, 2019; Gupta et al., 2020; Perez et al., 2020; Khot et al., 2021; Levine et al., 2022) for question-answering and other tasks.”
Learning to Solve Complex Tasks by Talking to Agents (2021, methodology): “Models using question decomposition (Talmor and Berant, 2018; Min et al., 2019b; Perez et al., 2020; Khot et al., 2021) are often dataset-speciﬁc, rely on decomposition annotations, and limited to one or two QA agents.”
F1 Is Not Enough! Models and Evaluation towards User-Centered Explainable Question Answering (2020, background): “Other approaches are based on question decomposition (Min et al., 2019; Perez et al., 2020), graph/hierarchical structures (Tu et al., 2019; Fang et al., 2019; Asai et al., 2020), virtual knowledge bases (Dhingra et al., 2020) or transformer models (Zhao et al., 2020).”
Solving math word problems with process- and outcome-based feedback (2022, unknown): “While the meanings of strong vs. weak supervision can vary depending on context, approaches which supervise intermediate steps are often referred to as strongly supervised (Yang et al., 2018; Perez et al., 2020).”
Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks (2021, background): “Several prior work attempts to learn to ﬁnd evidence sentences in unsupervised manners in multi-hop QA (Chen et al., 2019; Yadav et al., 2019; Perez et al., 2020), whereas our work uses evidentiality to improve the generator components via multi-task training for diverse knowledge-intensive tasks,…”
Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting (2023, background): “Decomposition-based approaches (Min et al., 2019; Perez et al., 2020; Chen et al., 2022; Creswell and Shanahan, 2022; Tafjord et al., 2022; Eisenstein et al., 2022; Reppert et al., 2023) improve faithfulness by limiting contextual cues that may bias CoT reasoning, with Radhakrishnan et al. (2023)…”
Rissanen Data Analysis: Examining Dataset Characteristics via Description Length (2021, methodology): “…time input modification has been used to evaluate the usefulness of the capability to decompose a question into subquestions (Min et al., 2019b; Perez et al., 2020), to access the image for image-based question-answering (Antol et al., 2015; Zhang et al., 2016a), and to view the premise when…”