Scaling question answering to the Web

By Cody C. T. Kwok, Oren Etzioni, Daniel S. Weld. 2001

607 citations recorded by the source corpus.

The wealth of information on the web makes it an attractive resource for seeking quick answers to simple, factual questions such as &quote;who was the first American in space?&quote; or &quote;what is the second tallest mountain in the world?&quote; Yet today's most advanced web search services (e.g., Google and AskJeeves) make it surprisingly tedious to locate answers to such questions. In this paper, we extend question-answering techniques, first studied in the information retrieval literature, to the web and experimentally evaluate their performance.First we introduce Mulder, which we believe to be the first general-purpose, fully-automated question-answering system available on the web. Second, we describe Mulder's architecture, which relies on multiple search-engine queries, natural-language parsing, and a novel voting procedure to yield reliable answers coupled with high recall. Finally, we compare Mulder's performance to that of Google and AskJeeves on questions drawn from the TREC-8 question answering track. We find that Mulder's recall is more than a factor of three higher than that of AskJeeves. In addition, we find that Google requires 6.6 times as much user effort to achieve the same level of recall as Mulder.

Subfields: Artificial Intelligence, Information Retrieval & Search, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Retrieval & Search

In Natural Language Processing

Citations in the corpus

Dense Passage Retrieval for Open-Domain Question Answering (2020, neutral): “Most QA systems in this era implemented a sophisticated pipeline approach or lever-aged the Web redundancy to extract frequently occurring phrases as answers (Brill et al., 2002; Kwok et al., 2001).”
Evidence Aggregation for Answer Re-Ranking in Open-Domain Question Answering (2017, background): “The task is to produce the answer to a question by exploiting resources such as documents (Voorhees, 1999), webpages (Kwok et al., 2001) or structured knowledge bases (Berant et al., 2013; Bordes et al., 2015; Yu et al., 2017).”
Progressively Pretrained Dense Corpus Index for Open-Domain Question Answering (2020, background): “Studies in the early stage (Kwok et al., 2001; Brill et al., 2002; Ferrucci et al., 2010; Baudiˇs, 2015) mostly rely on highly sophisticated pipelines and heterogeneous resources.”
Denoising Distantly Supervised Open-Domain Question Answering (2018, background): “, 1999), webpages (Kwok et al., 2001; Chen and Van Durme, 2017), structured knowledge graphs (Berant et al.”
Sparse, Dense, and Attentional Representations for Text Retrieval (2020, background): “Retrieving relevant documents is a core task for language technology, and is a component of applications such as information extraction and question answering (e.g., Narasimhan et al., 2016; Kwok et al., 2001; Voorhees, 2001).”
Open Question Answering with Weakly Supervised Embedding Models (2014, background): “The ﬁrst successful systems transformed the questions into queries which were fed to web search engines, the answer being subsequently extracted from top returned pages or snippets [13,1].”
Question Answering via Integer Programming over Semi-Structured Knowledge (2016, background): “Information Retrieval (IR) systems work under the assumption that answers to many questions of interest are often explicitly stated somewhere [Kwok et al., 2001], and all one needs, in principle, is access to a sufficiently large corpus.”
R3: Reinforced Reader-Ranker for Open-Domain Question Answering (2017, background): “The task is to produce the answer to a question by exploiting resources such as documents (Voorhees, 1999), webpages (Kwok et al., 2001) or structured knowledge bases (Berant et al., 2013; Bordes et al., 2015; Yu et al., 2017).”
An Analysis of the AskMSR Question-Answering System (2002, background): “Recently, other researchers have also looked to the web as a resource for question answering (Buchholtz, 2001; Clarke et al., 2001; Kwok et al., 2001).”
BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis (2019, background): “…of products and services, it is very hard, if not impossible, to pre-compile an up-to-date and reliable knowledge base to cover a wide assortment of questions that customers may ask, such as in factoid-based KB-QA (Xu et al., 2016; Fader et al., 2014; Kwok et al., 2001; Yin et al., 2015).”
Learning What is Essential in Questions (2017, neutral): “derive significant leverage from relatively shallow Information Retrieval (IR) and statistical correlation techniques operating on large unstructured corpora (Kwok et al., 2001; Clark et al., 2016).”
Factoid Question Answering over Unstructured and Structured Web Content (2005, background): “While previous approaches investigated how to scale current paradigms to general QA on the web (e.g. Kwok [11]), one of our main goals was to determine what performance can be achieved with a moderate annotation effort (in our case, one person-day) by a web-based QA system.”
Paraphrase-Driven Learning for Open Question Answering (2013, background): “More recently, researchers have created systems that use machine learning techniques to automatically construct question answering systems from data (Zelle and Mooney, 1996; Popescu et al., 2004; Zettlemoyer and Collins, 2005; Clarke et al., 2010; Liang et al., 2011).”
Open question answering over curated and extracted knowledge bases (2014, background): “In 4We discard Freebase facts that are not binary relations. contrast to the semantic parsing literature [30], a oqa query is not intended to represent the complete, formal semantic interpretation of a question.”
Enabling deep learning for large scale question answering in Italian (2019, neutral): “…processing that aims at extracting requirements and objectives of the user’s query, the retrieval phase where documents and sentences that include the answers are retrieved from the text collection and the answer extraction phase that locates the answer within the candidate sentences [7, 11].”
Autonomously semantifying wikipedia (2007, background): “MULDER [18], AskMSR [7], and KNOWITALL [14], showing some signs of early success.”
Natural Language Questions for the Web of Data (2012, methodology): “State-of-the-art methods (Hirschman and Gaizauskas, 2001; Kwok et al., 2001; Zheng, 2002; Katz et al., 2007; Dang et al., 2007; Voorhees, 2003) cast the user’s question into a keyword query to a Web search engine (perhaps with phrases for location and person names or other proper nouns).”