Break It Down: A Question Understanding Benchmark

By Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldberg, Daniel Deutch, Jonathan Berant. 2020

228 citations recorded by the source corpus.

Understanding natural language questions entails the ability to break down a question into the requisite steps for computing its answer. In this work, we introduce a Question Decomposition Meaning Representation (QDMR) for questions. QDMR constitutes the ordered list of steps, expressed through natural language, that are necessary for answering a question. We develop a crowdsourcing pipeline, showing that quality QDMRs can be annotated at scale, and release the Break dataset, containing over 83K pairs of questions and their QDMRs. We demonstrate the utility of QDMR by showing that (a) it can be used to improve open-domain question answering on the HotpotQA dataset, (b) it can be deterministically converted to a pseudo-SQL formal language, which can alleviate annotation in semantic parsing applications. Last, we use Break to train a sequence-to-sequence model with copying that parses questions into QDMR structures, and show that it substantially outperforms several natural baselines.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Dense Passage Retrieval for Open-Domain Question Answering (2020, methodology): “In practice, the sparse vector space model like TF-IDF or BM25 is a very strong baseline for retrieval and has been viewed as the standard method applied broadly to various QA tasks (e.g., Chen et al., 2017; Nie et al., 2019; Min et al., 2019a; Wolfson et al., 2020).”
Reasoning Like Program Executors (2022, background): “With a hybrid architecture, i.e., symbolic reasoners attached to LMs, neural-symbolic reasoning shines in a variety of reasoning tasks (Chen et al., 2020c; Tu et al., 2020; Wolfson et al., 2020).”
Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies (2021, methodology): “This issue has prompted recent work to create question decompositions for existing datasets (Wolfson et al., 2020), and to train models that generate question decompositions (Perez et al., 2020; Khot et al., 2020b; Min et al., 2019).”
Explainable Multi-hop Verbal Reasoning Through Internal Monologue (2021, methodology): “Question Decomposition: Our work is also different from several existing works about question decomposition, where the strategy of decomposition is largely reﬂected by the question itself (Min et al., 2019; Wolfson et al., 2020).”
KILT: a Benchmark for Knowledge Intensive Language Tasks (2020, background): “, 2019), question understanding (Wolfson et al., 2020), and dialogue (Shuster et al.”
Unsupervised Question Decomposition for Question Answering (2020, background): “More traditional, semantic parsing methods map questions to compositional programs, whose subprograms can be viewed as question decompositions in a formal language (Talmor and Berant, 2018; Wolfson et al., 2020).”
Learning To Retrieve Prompts for In-Context Learning (2021, background): “In this setting, we evaluate on the full development sets of B REAK , MT OP , and SMC AL F LOW .”
Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models (2021, methodology): “The BREAK dataset (Wolfson et al., 2020), on the other hand, provides the following high-level logical form for the first example:”
Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval (2020, methodology): “Wolfson et al. (2020) shows that with TF-IDF, using decomposed questions improves the retrieval results.”
QED: A Framework and Dataset for Explanations in Question Answering (2020, background): “…Jain and Wallace, 2019; Mohankumar et al., 2020) or sequential outputs (Camburu et al., 2018, 2019; Narang et al., 2020; Kumar and Talukdar, 2020) and more elaborate, discrete semantic representations that can in theory be applied to explainable QA (Abzianidze et al., 2017; Wolfson et al., 2020).”
Decomposed Prompting: A Modular Approach for Solving Complex Tasks (2022, methodology): “In our experiments we use two of the compositional operators defined by Khot et al. (2022) (see appendix for details), although it is capable of using all their operators (which also capture the QDMR operators from Wolfson et al. (2020)).”
Benchmarking Meaning Representations in Neural Semantic Parsing (2020, methodology): “Wolfson et al. (2020) designed an MR, named QDMR, for representing the meaning of questions through question decomposition.”
Obtaining Faithful Interpretations from Compositional Neural Networks (2020, methodology): “The input to this generation process is the set of crowdsourced question decompositions from the BREAK dataset (Wolfson et al., 2020).”
Is a Question Decomposition Unit All We Need? (2022, unknown): “Wolfson et al. (2020) crowd-sourced annotations for decompositions of questions.”
Case-based Reasoning for Natural Language Queries over Knowledge Bases (2021, background): “, 2010) to more recent systems for answering questions about text (Das et al., 2019; Min et al., 2019; Perez et al., 2020; Wolfson et al., 2020) or knowledge bases (Talmor & Berant, 2018).”
Making Retrieval-Augmented Language Models Robust to Irrelevant Context (2023, background): “3), shown to be necessary for high performance on multi-hop questions (Wolfson et al., 2020; Press et al., 2023).”
Cross-Policy Compliance Detection via Question Answering (2021, background): “Many researchers explored decomposing questions into simpler questions (Wolfson et al., 2020; Min et al., 2019; Fer-rucci et al., 2010; Perez et al., 2020).”