Interpretable & Time-Budget-Constrained Contextualization for Re-Ranking

By Sebastian Hofstätter, Markus Zlabinger, Allan Hanbury. 2020

84 citations recorded by the source corpus.

Search engines operate under a strict time constraint as a fast response is paramount to user satisfaction. Thus, neural re-ranking models have a limited time-budget to re-rank documents. Given the same amount of time, a faster re-ranking model can incorporate more documents than a less efficient one, leading to a higher effectiveness. To utilize this property, we propose TK (Transformer-Kernel): a neural re-ranking model for ad-hoc search using an efficient contextualization mechanism. TK employs a very small number of Transformer layers (up to three) to contextualize query and document word embeddings. To score individual term interactions, we use a document-length enhanced kernel-pooling, which enables users to gain insight into the model. TK offers an optimal ratio between effectiveness and efficiency: under realistic time constraints (max. 200 ms per query) TK achieves the highest effectiveness in comparison to BERT and other re-ranking models. We demonstrate this on three large-scale ranking collections: MSMARCO-Passage, MSMARCO-Document, and TREC CAR. In addition, to gain insight into TK, we perform a clustered query analysis of TK's results, highlighting its strengths and weaknesses on queries with different types of information need and we show how to interpret the cause of ranking differences of two documents by comparing their internal scores.

Subfields: Artificial Intelligence, Information Retrieval & Search, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Retrieval & Search

In Signal Processing

Citations in the corpus

Pretrained Transformers for Text Ranking: BERT and Beyond (2020, background): “, 2020a), and other transformer architectures, including groundup redesign efforts (Hofstätter et al., 2020b; Mitra et al., 2020) and adapting pretrained sequence-tosequence models (Nogueira et al.”
Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reduction (2022, methodology): “The max-sum operator scans the matrix of all term-by-term interactions, which is a technique inspired by earlier works on kernelpooling [18, 56].”
Expansion via Prediction of Importance with Contextualization (2020, methodology): “g signals from both a representation-based and an interaction-based model. However, this approach substantially under-performs the latest pure interaction-based approaches such as the one in [13]. TK [8] attempts to bridge this performance gap by using a smaller transformer network, but still utiliz”
Long Document Ranking with Query-Directed Sparse Transformer (2020, methodology): “• TK (Hofstätter et al., 2020) and TKL (Hofstätter et al., 2020) apply BERT-based kernels to estimate the relevance over document tokens with full attention.”
Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation (2020, methodology): “The Transformer-Kernel (TK) model [17] is not based on BERT pre-training, but rather uses shallow Transformers.”
Conformer-Kernel with Query Term Independence for Document Retrieval (2020, methodology): “This approach can not only negatively impact retrieval quality, but has been shown to specifically under-retrieve longer documents [Hofstätter et al., 2020a].”
Local Self-Attention over Long Text for Efficient Document Retrieval (2020, methodology): “h TK [11] 200 e0.”
SLEDGE: A Simple Yet Effective Baseline for Coronavirus Scientific Knowledge Search (2020, background): “This suggests that more effective utilization of approaches that include a count-based component in the ranking score (such as TK (Hofst¨atter et al., 2020) or CEDR-KNRM (MacAvaney et al., 2019)) could yield improvements.”
A Survey of Model Architectures in Information Retrieval (2025, unknown): “TK (Hofstätter et al., 2020c) uses a shallow transformer neural network (up to 3 layers) to encode the query Q and document D separately.”