Assessing efficiency–effectiveness tradeoffs in multi-stage retrieval systems without using relevance judgments

By C. Clarke, J. Shane Culpepper, Alistair Moffat. 2015

37 citations recorded by the source corpus.

Large-scale retrieval systems are often implemented as a cascading sequence of phases—a first filtering step, in which a large set of candidate documents are extracted using a simple technique such as Boolean matching and/or static document scores; and then one or more ranking steps, in which the pool of documents retrieved by the filter is scored more precisely using dozens or perhaps hundreds of different features. The documents returned to the user are then taken from the head of the final ranked list. Here we examine methods for measuring the quality of filtering and preliminary ranking stages, and show how to use these measurements to tune the overall performance of the system. Standard top-weighted metrics used for overall system evaluation are not appropriate for assessing filtering stages, since the output is a set of documents, rather than an ordered sequence of documents. Instead, we use an approach in which a quality score is computed based on the discrepancy between filtered and full evaluation. Unlike previous approaches, our methods do not require relevance judgments, and thus can be used with virtually any query set. We show that this quality score directly correlates with actual differences in measured effectiveness when relevance judgments are available. Since the quality score does not require relevance judgments, it can be used to identify queries that perform particularly poorly for a given filter. Using these methods, we explore a wide range of filtering options using thousands of queries, categorize the relative merits of the different approaches, and identify useful parameter combinations.

Subfields: Artificial Intelligence, Information Systems, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Management Science and Operations Research

Citations in the corpus

Anserini (2018, background): “Although we have not explicitly addressed this (next) challenge, we believe that baseline bag-of-words ranking models also have an important role to play in modern multi-stage ranking architectures, which have been explored by researchers [8, 16, 17, 36, 49, 55] and deployed commercially [33, 43].”
Anserini: Enabling the Use of Lucene for Information Retrieval Research (2017, background): “In particular, we advocate a multi-stage ranking architecture [3, 6, 14, 17] so that researchers will not need to directly work with native Lucene scoring APIs. is, researchers should take advantage of Anserini APIs that generate an initial document ranking and hooks for feature extraction to build…”
Query Driven Algorithm Selection in Early Stage Retrieval (2017, methodology): “2, based on other work that shows that more aggressive approaches result in reduced effectiveness [13].”