The Web as a Knowledge-Base for Answering Complex Questions
By Alon Talmor, Jonathan Berant. 2018
832 citations recorded by the source corpus.
Answering complex questions is a time-consuming activity for humans that requires reasoning and integration of information. Recent work on reading comprehension made headway in answering simple questions, but tackling complex questions is still an ongoing research challenge. Conversely, semantic parsers have been successful at handling compositionality, but only when the information resides in a target knowledge-base. In this paper, we present a novel framework for answering broad and complex questions, assuming answering simple questions is possible using a search engine and a reading comprehension model. We propose to decompose complex questions into a sequence of simple questions, and compute the final answer from the sequence of answers. To illustrate the viability of our approach, we create a new dataset of complex questions, ComplexWebQuestions, and present a model that decomposes questions and interacts with the web to compute an answer. We empirically demonstrate that question decomposition improves performance from 20.8 precision@1 to 27.5 precision@1 on this new dataset.
Subfields: Artificial Intelligence, Natural Language Processing, Question Answering & Reading Comprehension
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Percy Liang
- Hannaneh Hajishirzi
- Jianfeng Gao
- Yejin Choi
- Graham Neubig
- Noah A. Smith
In Natural Language Processing
- Mohit Bansal
- Hannaneh Hajishirzi
- Luke S. Zettlemoyer
- Yejin Choi
- Percy Liang
- Jonathan Berant
- Graham Neubig
- Noah A. Smith
In Question Answering & Reading Comprehension
- Ashish Sabharwal
- Tushar Khot
- Peter Clark
- William W. Cohen
- Haitian Sun
- Hannaneh Hajishirzi
- Kyle Richardson
- Jonathan Berant
Citations in the corpus
- DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (2019, background): “, 2018), requiring passage retrieval (Joshi et al., 2017; Yang et al., 2018; Talmor and Berant, 2018), mismatched passages and questions (Saha et al.”
- CoQA: A Conversational Question Answering Challenge (2018, background): “…a hand-written grammar to create artificial questions (Weston et al., 2016; Welbl et al., 2018), paraphrasing artificial questions to natural questions (Saha et al., 2018; Talmor and Berant, 2018), or, in our case, letting humans ask natural questions (Rajpurkar et al., 2016; Nguyen et al., 2016).”
- QuAC: Question Answering in Context (2018, background): “CoQA (Reddy et al., 2018) CSQA (Saha et al., 2018) CQA (Talmor and Berant, 2018) SQA (Iyyer et al., 2017) NarrativeQA (Kocisk´y et al., 2017) TriviaQA (Joshi et al., 2017) SQuAD 2.0 (Rajpurkar et al., 2018) MS Marco (Nguyen et al., 2016) NewsQA (Trischler et al., 2016) Table 1: Comparison of the QUA”
- Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies (2021, background): “• T WENTY Q UESTIONS (20Q): A collection of 50K short commonsense boolean questions.”
- SPoC: Search-based Pseudocode to Code (2019, background): “While semantic parsing datasets with shorter, query-like programs [57, 7, 33, 4, 59, 23, 44, 55] usually have an execution environment (e.”
- Augmented Language Models: a Survey (2023, background): “While many earlier works include learning to decompose through distant supervision (Perez et al., 2020; Talmor and Berant, 2018; Min et al., 2019), like Zhou et al. (2022), many recent works employ in-context learning to do so (Yang et al., 2022a; Khot et al., 2022; Dua et al., 2022; Kazemi et al.,…”
- Break It Down: A Question Understanding Benchmark (2020, background): “This includes reasoning over images (Antol et al., 2015; Johnson et al., 2017; Suhr et al., 2019; Hudson and Manning, 2019), paragraphs (Dua et al., 2019), documents (Welbl et al., 2018; Talmor and Berant, 2018; Yang et al., 2018), tables (Pasupat and Liang, 2015) and more.”
- oLMpics-On What Language Model Pre-training Captures (2019, background): “, 2019b) and facilitates probing question involving complex and commonsense reasoning (Talmor et al., 2019; Gardner et al., 2019a; Talmor and Berant, 2018).”
- The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpus (2021, background): “Previous works that operate on web (Joshi et al., 2017; Bajaj et al., 2018; Talmor and Berant, 2018) typically rely on results from black-box search engines to create a corpus.”
- HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering (2018, methodology): “, 2018) and COMPLEXWEBQUESTIONS (Talmor and Berant, 2018) explore different approaches of using pre-existing”
- UL2: Unifying Language Learning Paradigms (2022, methodology): “We use several component tasks from UnifiedSKG (Xie et al., 2022), namely WikiTQ (Pasupat & Liang, 2015), CompWQ (Talmor & Berant, 2018), FetaQA (Nan et al., 2021), HybridQA (Chen et al., 2020), WikiSQL (Zhong et al., 2017), TabFat (Chen et al., 2019), Feverous (Aly et al., 2021), SQA (Iyyer et al.,”
- CommonsenseQA 2.0: Exposing the Limits of AI through Gamification (2021, background): “Our work joins a line of previously introduced benchmarks targeting commonsense reasoning [12, 13, 29, 15] as well as probing for other reasoning abilities [32, 33, 34, 35, 36, 37, 38].”
- KILT: a Benchmark for Knowledge Intensive Language Tasks (2020, methodology): “…(Clark et al., 2018), TriviaQA-web (Joshi et al., 2017), Quasar-T (Dhingra et al., 2017), WebQuestions (Berant et al., 2013) and ComplexWebQuestions (Talmor and Berant, 2018)—in KILT we exclusively consider publicly available Wikipedia-based datasets in order to merge and unify the knowledge…”
- Unsupervised Question Decomposition for Question Answering (2020, background): “, 2017b) and knowledge-base QA (Talmor and Berant, 2018), as well as claims in fact verification (Thorne et al.”
- UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022, background): “‡ We use gold entity linking, but the previous sota does not, which makes the results not directly comparable; therefore, we do not bold any numbers for CompWebQ and HybridQA.”
- Neural Approaches to Conversational AI (2018, neutral): “x question Q, we decompose it to a sequence of simple questions Q 1;Q 2;:::, use a Web-scale KB-QA agent to generate for each Q ian answer A i, from which we compute the final answer A. Figure credit: Talmor and Berant (2018). Given a user’s description of a problem e.g., “cannot update the personal ”
- QASC: A Dataset for Question Answering via Sentence Composition (2019, methodology): “In fact, multi-hop questions in many existing datasets (Yang et al. 2018; Talmor and Berant 2018) often follow this syntactically decomposable pattern, with questions such as: “Which government position was held by the lead actress of X?””
- Compositional Questions Do Not Necessitate Multi-hop Reasoning (2019, methodology): “To foster research on reasoning across multiple pieces of text, multi-hop QA has been introduced (Koˇcisk`y et al., 2018; Talmor and Berant, 2018; Yang et al., 2018).”
- Unsupervised Commonsense Question Answering with Self-Talk (2020, background): “…machine reading (Saeidi et al., 2018; Pan et al., 2019), simplifying questions to make them more easily answerable (Buck et al., 2018; Talmor and Berant, 2018; Perez et al., 2020), to us-ing questions as means for other purposes such as sentence representation and summarization (Guo…”
- MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension (2019, background): “ComplexWebQ (Talmor and Berant, 2018) ComplexWebQuestions is collected by crowdworkers who are shown compositional, formal queries against Freebase, and are asked to rephrase them in natural language.”