Local Self-Attention over Long Text for Efficient Document Retrieval

By Sebastian Hofstätter, Hamed Zamani, Bhaskar Mitra, Nick Craswell, Allan Hanbury. 2020

96 citations recorded by the source corpus.

Neural networks, particularly Transformer-based architectures, have achieved significant performance improvements on several retrieval benchmarks. When the items being retrieved are documents, the time and memory cost of employing Transformers over a full sequence of document terms can be prohibitive. A popular strategy involves considering only the first n terms of the document. This can, however, result in a biased system that under retrieves longer documents. In this work, we propose a local self-attention which considers a moving window over the document terms and for each term attends only to other terms in the same window. This local attention incurs a fraction of the compute and memory cost of attention over the whole document. The windowed approach also leads to more compact packing of padded documents in minibatches resulting in additional savings. We also employ a learned saturation function and a two-staged pooling strategy to identify relevant regions of the document. The Transformer-Kernel pooling model with these changes can efficiently elicit relevance information from documents with thousands of tokens. We benchmark our proposed modifications on the document ranking task from the TREC 2019 Deep Learning track and observe significant improvements in retrieval quality as well as increased retrieval of longer documents at moderate increase in compute and memory costs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Pretrained Transformers for Text Ranking: BERT and Beyond (2020, neutral): “372 (2a) Co-PACRR [Hofstätter et al., 2020] 0.”
Long Document Ranking with Query-Directed Sparse Transformer (2020, methodology): “• TK (Hofstätter et al., 2020) and TKL (Hofstätter et al., 2020) apply BERT-based kernels to estimate the relevance over document tokens with full attention.”
Conformer-Kernel with Query Term Independence for Document Retrieval (2020, methodology): “This approach can not only negatively impact retrieval quality, but has been shown to specifically under-retrieve longer documents [Hofstätter et al., 2020a].”
BERT-QE: Contextualized Query Expansion for Document Re-ranking (2020, background): “Many efforts have also been made to reduce the inference time of transformer-based rankers (Hofst¨atter et al., 2020a,b; Gao et al., 2020).”
Do Long-Range Language Models Actually Use Long-Range Context? (2021, unknown): “Hofstätter et al., 2020; Zhang et al., 2020).”
Learning Passage Impacts for Inverted Indexes (2021, methodology): “To measure effectiveness, we use the official metrics for each query set, mean reciprocal rank (MRR@10) for MSMARCO queries, and normalised discounted cumulative gain (NDCG@10) as well as mean average precision (MAP) for TREC queries, following [8].”
Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation (2020, methodology): “Several approaches aim to improve the efficiency of transformer models with windowed self-attention [16], using locality-sensitive hashing [22], replacing the self-attention with a local windowed and global attention [2] or by combining an efficient transformer-kernel model with a conformer layer…”
Improving the Prosody of RNN-Based English Text-To-Speech Synthesis by Incorporating a BERT Model (2020, background): “This is different from local attention [19, 20, 21] as the Transformer architecture itself is unaltered in our case.”
Relative Positional Encoding for Transformers with Linear Complexity (2021, unknown): “Further comparisons with local attention schemes (e.g. Dai et al., 2019; Hofst¨atter et al., 2020) could be interesting, although they were not included here due to Tay et al. (2021) suggesting that they are clearly inferior, at least in the LRA setting.”
A Modern Perspective on Query Likelihood with Deep Generative Retrieval Models (2021, unknown): “Arguably, all existing neural ranking models [7, 11, 12, 14, 25, 28, 33, 50] follow the text matching paradigm, where relevance is calculated as the predicted matching score of each document to a given query.”
A Survey of Model Architectures in Information Retrieval (2025, unknown): “TK (Hofstätter et al., 2020c) uses a shallow transformer neural network (up to 3 layers) to encode the query Q and document D separately.”