Query Driven Algorithm Selection in Early Stage Retrieval

By Joel M. Mackenzie, J. Shane Culpepper, Roi Blanco, Matt Crane, C. Clarke, Jimmy J. Lin. 2017

65 citations recorded by the source corpus.

Scalable web search systems typically employ multi-stage retrieval architectures, where an initial stage generates a set of candidate documents that are then pruned and re-ranked. Since subsequent stages typically exploit a multitude of features of varying costs using machine-learned models, reducing the number of documents that are considered at each stage improves latency. In this work, we propose and validate a unified framework that can be used to predict a wide range of performance-sensitive parameters which minimize effectiveness loss, while simultaneously minimizing query latency, across all stages of a multi-stage search architecture. Furthermore, our framework can be easily applied in large-scale IR systems, can be trained without explicitly requiring relevance judgments, and can target a variety of different efficiency-effectiveness trade-offs, making it well suited to a wide range of search scenarios. Our results show that we can reliably predict a number of different parameters on a per-query basis, while simultaneously detecting and minimizing the likelihood of tail-latency queries that exceed a pre-specified performance budget. As a proof of concept, we use the prediction framework to help alleviate the problem of tail-latency queries in early stage retrieval. On the standard ClueWeb09B collection and 31k queries, we show that our new hybrid system can reliably achieve a maximum query time of 200 ms with a 99.99% response time guarantee without a significant loss in overall effectiveness. The solutions presented are practical, and can easily be used in large-scale distributed search engine deployments with a small amount of additional overhead.

Subfields: Computer Vision and Pattern Recognition, Information Systems, Information Retrieval & Search, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Information Systems

In Information Retrieval & Search

In Signal Processing

Citations in the corpus

Pretrained Transformers for Text Ranking: BERT and Beyond (2020, background): “age ranking architectures have received substantial interest in the academic literature [Tonellotto et al., 2013, Asadi and Lin, 2013, Capannini et al., 2016, Clarke et al., 2016, Chen et al., 2017b, Mackenzie et al., 2018] as well as industry. Among publicly documented deployments at scale, beyond ”
Multi-Stage Document Ranking with BERT (2019, background): “This basic approach has received much interest in academia (Matveeva et al., 2006; Wang et al., 2011; Asadi and Lin, 2013; Chen et al., 2017; Mackenzie et al., 2018) as well as industry.”