Efficient Document Re-Ranking for Transformers by Precomputing Term Representations

By Sean MacAvaney, Franco Maria Nardini, Raffaele Perego, Nicola Tonellotto, Nazli Goharian, Ophir Frieder. 2020

132 citations recorded by the source corpus.

Deep pretrained transformer networks are effective at various ranking tasks, such as question answering and ad-hoc document ranking. However, their computational expenses deem them cost-prohibitive in practice. Our proposed approach, called PreTTR (Precomputing Transformer Term Representations), considerably reduces the query-time latency of deep transformer networks (up to a 42x speedup on web document ranking) making these networks more practical to use in a real-time ranking scenario. Specifically, we precompute part of the document term representations at indexing time (without a query), and merge them with the query representation at query time to compute the final ranking score. Due to the large size of the token representations, we also propose an effective approach to reduce the storage requirement by training a compression layer to match attention scores. Our compression technique reduces the storage required up to 95% and it can be applied without a substantial degradation in ranking performance.

Subfields: Artificial Intelligence, Information Systems, Management Science and Operations Research, Information Retrieval & Search, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Management Science and Operations Research

In Information Retrieval & Search

In Large Language Models

Citations in the corpus

Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval (2020, background): “Their efficiency also becomes a concern when pretrained models are used [37, 38].”
You Only Need One Model for Open-domain Question Answering (2021, neutral): “Similar to our approach of re-ranker on top of a shared retriever, PreTTR (MacAvaney et al., 2020) pre-computed term representations for all documents, and used these to run only the upper layers of a transformer reranker model.”
Learning Passage Impacts for Inverted Indexes (2021, methodology): “PreTTR [12] and ColBERT [9] experimentally show that the query-document interactions in most layers of BERT have little impact on the final effectiveness.”
BERT-QE: Contextualized Query Expansion for Document Re-ranking (2020, methodology): “In future work, we plan to further improve the efﬁciency of BERT-QE, by combining the proposed BERT-QE with the the pre-computation techniques proposed recently (Khattab and Zaharia, 2020; MacAvaney et al., 2020a), wherein most of the computation could be performed ofﬂine.”
DiPair: Fast and Accurate Distillation for Trillion-ScaleText Matching and Pair Modeling (2020, methodology): “Unlike the models proposed in the recent works (MacAvaney et al., 2020; Humeau et al., 2020) where only pairs can be supported, our proposed architecture trivially extends to the scenario where we have n-ary tuple of textual objects (a1, a2, .”
Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation (2020, methodology): “The PreTT architecture [27] is conceptually between BERTCAT and ColBERT, as it allows to compute b BERT-layers separately for query and passage:”
SLEDGE: A Simple Yet Effective Baseline for Coronavirus Scientific Knowledge Search (2020, background): “…more efﬁcient modeling choices (Hofst¨atter et al., 2020; MacAvaney et al., 2020c), caching intermediate representations (Khattab and Zaharia, 2020; MacAvaney et al., 2020b; Gao et al., 2020), or by modifying the index with new terms or weights (Nogueira et al., 2019; Dai and Callan, 2019a;…”
COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted List (2021, background): “Lighter deep LM rankers are developed (MacAvaney et al., 2020; Gao et al., 2020), but their cross attention operations are still too expensive for fullcollection retrieval.”
A Modern Perspective on Query Likelihood with Deep Generative Retrieval Models (2021, unknown): “…of frequencies across ranges of similarity values [7, 12–14 , For instance, the BERT model is used for document/passage retrieval through fine-tuning [31], combining them with other ranking models [27], expanding to other more efficient variations [26] or dense retrieval approaches [22, 51].”