An Analysis of the AskMSR Question-Answering System

By Eric Brill, Susan T. Dumais, Michele Banko. 2002

366 citations recorded by the source corpus.

We describe the architecture of the AskMSR question answering system and systematically evaluate contributions of different system components to accuracy. The system differs from most question answering systems in its dependency on data redundancy rather than sophisticated linguistic analyses of either questions or candidate answers. Because a wrong answer is often worse than no answer, we also explore strategies for predicting when the question answering system is likely to give an incorrect answer.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

SQuAD: 100,000+ Questions for Machine Comprehension of Text (2016, background): “One key difference between our RC setting and answer extraction is that answer extraction typically exploits the fact that the answer occurs in multiple documents (Brill et al., 2002), which is more lenient than in our setting, where a system only has access to a single reading passage.”
REALM: Retrieval-Augmented Language Model Pre-Training (2020, methodology): “Many of these systems employ a retrieval-based approach: given a question x, retrieve potentially relevant documents z from the corpus Z , and then extract an answer y from the documents (Brill et al., 2002; Chen et al., 2017; Lee et al., 2019).”
Reading Wikipedia to Answer Open-Domain Questions (2017, methodology): “There are a number of highly developed full pipeline QA approaches using either the Web, as does QuASE (Sun et al., 2015), or Wikipedia as a resource, as do Microsoft’s AskMSR (Brill et al., 2002), IBM’s DeepQA (Ferrucci et al., 2010) and YodaQA (Baudi ˇ s, 2015; Baudi ˇ s and ˇSediv ` y, 2015) — th”
Progressively Pretrained Dense Corpus Index for Open-Domain Question Answering (2020, background): “Studies in the early stage (Kwok et al., 2001; Brill et al., 2002; Ferrucci et al., 2010; Baudiˇs, 2015) mostly rely on highly sophisticated pipelines and heterogeneous resources.”
Combining Retrieval, Statistics, and Inference to Answer Elementary Science Questions (2016, background): “…to short, factoid questions (e.g., “In which year was Bill Clinton born?”) by locating answers in databases (Yao and Van Durme 2014; Zou et al. 2014; Fader, Zettlemoyer, and Etzioni 2014) or large document collections (Brill, Dumais, and Banko 2002; Ferrucci et al. 2010; Ko, Nyberg, and Si 2007).”
Question Answering via Integer Programming over Semi-Structured Knowledge (2016, methodology): “…the capital of France?”) using document collections or databases (e.g., Freebase [Bollacker et al. , 2008], NELL [Carlson et al. , 2010]), for example [Brill et al. , 2002; Fader et al. , 2014; Ferrucci et al. , 2010; Ko et al. , 2007; Yih et al. , 2014; Yao and Van Durme, 2014; Zou et al. , 2014].”
Answering Complex Questions Using Open Information Extraction (2017, background): “Web QA: There exist several systems for retrieval-based Web QA problems (Ferrucci et al., 2010; Brill et al., 2002).”
Learning executable semantic parsers for natural language understanding (2016, background): “Even question answering systems relied less on understanding and more on a shallower analysis coupled with a large collection of unstructured text documents [10], typiﬁed by the TREC competitions.”
Web question answering: is more always better? (2002, background): “[5] have started to explore individual components in more detail, with interesting results.”
Open-Domain Question Answering (2020, background): “Open-domain question answering (QA), the task of answering questions using a large collection of documents of diversified topics, has been a longstanding problem in NLP, information retrieval (IR) and related fields (Voorhees et al., 1999; Moldovan et al., 2000; Brill et al., 2002; Ferrucci et al., ”
Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering (2021, methodology): “Following this pipeline, a well-known QA system AskMSR [44] was developed, which mainly depends on data redundancy rather than sophisticated linguistic analysis of either questions or candidate answers.”
Paraphrase-Driven Learning for Open Question Answering (2013, background): “There has been some work on QA over web text (Kwok et al., 2001; Brill et al., 2002), but these systems do not operate over extracted relational data.”
Modeling of the Question Answering Task in the YodaQA System (2015, background): “5% AskMSR [4] (web-search system) 61.”
Open Domain Question Answering via Semantic Enrichment (2015, methodology): “Prior to the availability and popularity of knowledge bases, most of early QA systems such as [7, 12, 25, 20, 37, 41], mine answers from TREC [41] document collections or the rich web corpus.”
Retrieve-and-Read: Multi-task Learning of Information Retrieval and Reading Comprehension (2018, methodology): “ IR, and answer extraction [17]. The systems reformulate queries toenable their IR method to cover many textualvariants.However, theirreformulationisdependent onthe redundancy of the knowledge source [4], and thus, they do not work well on smaller corpora. A deeper understanding of natural language ”
Using Semantic Roles to Improve Question Answering (2007, methodology): “We also reformulate questions into declarative sentences following the strategy proposed in Brill et al. (2002).”