Let's measure run time! Extending the IR replicability infrastructure to include performance aspects

By Sebastian Hofstätter, Allan Hanbury. 2019

48 citations recorded by the source corpus.

Establishing a docker-based replicability infrastructure offers the community a great opportunity: measuring the run time of information retrieval systems. The time required to present query results to a user is paramount to the users satisfaction. Recent advances in neural IR re-ranking models put the issue of query latency at the forefront. They bring a complex trade-off between performance and effectiveness based on a myriad of factors: the choice of encoding model, network architecture, hardware acceleration and many others. The best performing models (currently using the BERT transformer model) run orders of magnitude more slowly than simpler architectures. We aim to broaden the focus of the neural IR community to include performance considerations -- to sustain the practical applicability of our innovations. In this position paper we supply our argument with a case study exploring the performance of different neural re-ranking models. Finally, we propose to extend the OSIRRC docker-based replicability infrastructure with two performance focused benchmark scenarios.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Hardware and Architecture, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Hardware and Architecture

In Evaluation & Benchmarks

Citations in the corpus

ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT (2020, methodology): “For the baselines, we measure the scoring computations on the GPU and exclude the CPU-based text preprocessing (similar to [10]).”
Pretrained Transformers for Text Ranking: BERT and Beyond (2020, methodology): “Rows (2a) and (2b) are taken directly from Hofstätter et al. [2020] for Co-PACRR [Hui et al.”
Efficient Document Re-Ranking for Transformers by Precomputing Term Representations (2020, background): “While the efficiency of learning to rank solutions for document re-ranking have been extensively studied [6, 24, 41], computational efficiency concerns have largely be ignored by prior work in neural ranking, prompting some to call for more attention to this matter [16].”
Distilling Dense Representations for Ranking using Tightly-Coupled Teachers (2020, background): “The standard reranker architecture, while effective, exhibits high query latency, on the order of seconds per query (Hofstätter and Hanbury, 2019; Khattab and Zaharia, 2020) because expensive neural inference must be applied at query time on query–document pairs.”
Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling (2021, background): “This architecture is easy to train and provides very strong results in terms of effectiveness, especially when used in an ensemble [13], however it requires a candidate selection prior to re-ranking, has no ability to pre-compute and index passage representations and is therefore slow in practice [1”
Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation (2020, methodology): “BERTCAT (q1:m, p1:n) = BERT( [CLS;q1:m ; SEP;p1:n])1 ∗Ws (1) We utilize BERTCAT as our teacher architecture, as it represents the current state-of-the art in terms of effectiveness, however it requires substantial compute at query time and increases the query latency by seconds [15, 42].”
The neural hype, justified! (2019, background): “This was self-evident from the very beginning, and in the context of search, long query latencies for BERT-based models are well documented [13, 8].”
Interpretable & Time-Budget-Constrained Contextualization for Re-Ranking (2020, methodology): “[12] establish efficiency baselines for common neural IR models (including BERT) and propose to incorporate speed metrics in replicability campaigns and public leaderboards.”
Expansion via Prediction of Importance with Contextualization (2020, background): “However, these approaches have a substantial computational burden, which can make them impractical [7, 14].”
In-Batch Negatives for Knowledge Distillation with Tightly-Coupled Teachers for Dense Retrieval (2021, background): “The standard reranker architecture, while effective, exhibits high query latency, on the order of seconds per query (Hofstätter and Hanbury, 2019; Khattab and Zaharia, 2020) because expensive neural inference must be applied at query time on query–passage pairs.”