Quantitative evaluation of passage retrieval algorithms for question answering

By Stefanie Tellex, Boris Katz, Jimmy J. Lin, Aaron Fernandes, Gregory A. Marton. 2003

355 citations recorded by the source corpus.

Passage retrieval is an important component common to many question answering systems. Because most evaluations of question answering systems focus on end-to-end performance, comparison of common components becomes difficult. To address this shortcoming, we present a quantitative evaluation of various passage retrieval algorithms for question answering, implemented in a framework called Pauchok. We present three important findings: Boolean querying schemes perform well in the question answering task. The performance differences between various passage retrieval algorithms vary with the choice of document retriever, which suggests significant interactions between document retrieval and passage retrieval. The best algorithms in our evaluation employ density-based measures for scoring query terms. Our results reveal future directions for passage retrieval and question answering.

Subfields: Artificial Intelligence, Information Systems, Information Retrieval & Search, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Information Retrieval & Search

In Natural Language Processing

Citations in the corpus

GloVe: Global Vectors for Word Representation (2014, background): “These vectors can be used as features in a variety of applications, such as information retrieval (Manning et al., 2008), document classiﬁcation (Sebastiani, 2002), question answering (Tellex et al., 2003), named entity recognition (Turian et al., 2010), and parsing (Socher et al., 2013).”
End-to-End Open-Domain Question Answering with BERTserini (2019, background): “…retrieval, it was generally envisioned that a QA system would comprise pipeline stages that selected increasingly ﬁner-grained segments of text (Tellex et al., 2003): document retrieval to identify relevant documents from a large corpus, followed by passage ranking to identify text segments…”
From Frequency to Meaning: Vector Space Models of Semantics (2010, methodology): “In Turney’s (2007) tensor, for example, rows correspond to words from the TOEFL multiple-choice synonym questions, columns correspond to words from Basic English (Ogden, 1930),12 and tubes correspond to patterns that join rows and columns (hence we have a word–word–pattern third-order tensor).”
Is searching full text more effective than searching abstracts? (2009, background): “The problem of retrieving sub-document segments of text has previously been explored in the information retrieval literature [15-24]; the task is often called passage retrieval.”
Comparative study on methods of detecting research fronts using different types of citation (2009, methodology): “Tellex et al. (2003) compared seven passageretrieval algorithms such as Okapi BM-25 (Robertson et al., 2000; Savoy, 2005) and SiteQ (G.G.”
An Exploration of Approaches to Integrating Neural Reranking Models in Multi-Stage Ranking Architectures (2017, neutral): “In this paper, we focus on the question answering task, although the architecture is exactly the same: we consider a standard pipeline architecture [17] where the natural language question is first used as a query to retrieve a set of candidate documents, which are then segmented into sentences and…”
Improving Word Representations via Global Context and Multiple Word Prototypes (2012, background): “, 2008), document classification (Sebastiani, 2002) and question answering (Tellex et al., 2003).”
Rank learning for factoid question answering with linguistic and semantic constraints (2010, background): “Many passage retrieval approaches commonly used in QA systems can not check these types of constraints at query time.”
ITNLP-AiKF at SemEval-2017 Task 1: Rich Features Based SVR for Semantic Textual Similarity Computing (2017, background): “Word Embedding maps words or phrases from de-ﬁned vocabulary with dense vectors of real values, which have been applied as features in document classiﬁcation (Sebastiani, 2002), question answering (Tellex et al., 2003), and named entity recognition (Turian et al., 2010) tasks.”
Retrieve-and-Read: Multi-task Learning of Information Retrieval and Reading Comprehension (2018, neutral): “rQA:oneistoindex each passage as a separate document and retrieve them; the other one is to retrieve long documents from a corpus ﬁrst and then ﬁnd relevant short passages from theretrieved documents [44].Exploring the potential of such a two-stage IR in an end-to-end neural network model would be w”
Automatically Extracting High-Quality Negative Examples for Answer Selection in Question Answering (2017, methodology): “In a standard pipeline architecture [12], answer selection is applied to the output of a module that performs passage retrieval, typically using lightweight term-based matching.”