From Neural Re-Ranking to Neural Ranking: Learning a Sparse Representation for Inverted Indexing

By Hamed Zamani, Mostafa Dehghani, W. Bruce Croft, Eric Learned-Miller, Jaap Kamps. 2018

213 citations recorded by the source corpus.

The availability of massive data and computing power allowing for effective data driven neural approaches is having a major impact on machine learning and information retrieval research, but these models have a basic problem with efficiency. Current neural ranking models are implemented as multistage rankers: for efficiency reasons, the neural model only re-ranks the top ranked documents retrieved by a first-stage efficient ranker in response to a given query. Neural ranking models learn dense representations causing essentially every query term to match every document term, making it highly inefficient or intractable to rank the whole collection. The reliance on a first stage ranker creates a dual problem: First, the interaction and combination effects are not well understood. Second, the first stage ranker serves as a "gate-keeper" or filter, effectively blocking the potential of neural models to uncover new relevant documents. In this work, we propose a standalone neural ranking model (SNRM) by introducing a sparsity property to learn a latent sparse representation for each query and document. This representation captures the semantic relationship between the query and documents, but is also sparse enough to enable constructing an inverted index for the whole collection. We parameterize the sparsity of the model to yield a retrieval model as efficient as conventional term based models. Our model gains in efficiency without loss of effectiveness: it not only outperforms the existing term matching baselines, but also performs similarly to the recent re-ranking based neural models with dense representations. Our model can also take advantage of pseudo-relevance feedback for further improvements. More generally, our results demonstrate the importance of sparsity in neural IR models and show that dense representations can be pruned effectively, giving new insights about essential semantic features and their distributions.

Subfields: Artificial Intelligence, Information Systems, Information Retrieval & Search, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Information Retrieval & Search

In Model Architectures

Citations in the corpus

Pretrained Transformers for Text Ranking: BERT and Beyond (2020, background): “In fact, ranking with dense representations predates BERT by many years [6, 11, 15, 29, 38, 41].”
Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrieval (2019, background): “A second approach is to learn high-dimensional but sparse latent representations in which queries and documents are represented by a set of ‘latent words’ [28, 34].”
Multi-Stage Document Ranking with BERT (2019, background): “…where the task is to learn some non-linear transformation of queries and documents (i.e., using a neural network) such that documents relevant to a query have high similarities in terms of a simple metric such as cosine similarity (Hen-derson et al., 2017; Zamani et al., 2018; Ji et al., 2019).”
Multi-step Entity-centric Information Retrieval for Multi-Hop Question Answering (2019, neutral): “Neural ranking models have shown great potential and have been widely adopted in the IR community (Dehghani et al., 2017; Guo et al., 2019; Mitra et al., 2017; Zamani et al., 2018, inter-alia).”
Efficiency Implications of Term Weighting for Passage Retrieval (2020, background): “As such, there is growing interest in utilizing pre-trained models to enhance existing components of the retrieval pipeline beyond document reranking [5, 20, 26].”
SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking (2021, methodology): “This differs from the l1 regularization used in SNRM [28] where the a j are not squared: using lFLOPS thus pushes down high average term weight values, giving rise to a more balanced index.”
Critically Examining the "Neural Hype": Weak Baselines and the Additivity of Effectiveness Gains from Neural Ranking Models (2019, unknown): “st collection. Focusing specifically on the neural approaches, 8 out of 16 papers (50.0%) used a baseline that is below the TREC median. The highest reported score we encountered was by Zamani et al. [25] in 2018, at 0.2971. While there does appear to be an upward trend in the effectiveness of the n”
Conformer-Kernel with Query Term Independence for Document Retrieval (2020, methodology): “, 2019, Khattab and Zaharia, 2020] or by employing other data structures, such as learning an inverted index based on latent representations [Zamani et al., 2018].”
A Deep Look into Neural Ranking Models for Information Retrieval (2019, background): “However, the PRF technique can also be leveraged to enhance neural ranking models (e.g., SNRM+PRF [28] and NPRF+DRMM [119] in Table 1), while human designed LTR features can be integrated into neural ranking models [33, 31] to improve the ranking performance.”
Incorporating Query Term Independence Assumption for Efficient Retrieval and Ranking using Deep Neural Networks (2019, background): “An alternative involves learning higher dimensional but sparse representations of query and document [Salakhutdinov and Hinton, 2009, Zamani et al., 2018a] that can also be employed for fast lookup.”
On the Effect of Low-Frequency Terms on Neural-IR Models (2019, methodology): “els. OtherstudiesexplorerelatedaspectsofneuralIRmodels.Pyreddy et al. [12] investigate the variance and consistency of kernel-based neural models over various parameter initializations. Zamani et al. [18] propose a method to skip the re-ranking step, and directly retrieve documents from an index of ”
Cross-Domain Sentence Modeling for Relevance Transfer with BERT (2019, background): “[75] propose a standalone neural ranking model to learn a latent representation for each query and document, which is sparse enough to construct an inverted index for the entire collection.”
A Survey of the Usages of Deep Learning for Natural Language Processing (2020, background): “[170] presented a standalone neural ranking model called SNRM_PRF that learned sparse representations for both queries and documents, mimicking what traditional approaches do.”
Contextualized Sparse Representations for Real-Time Open-Domain Question Answering (2019, methodology): “more ﬁne-grained than paragraphs or documents. There have been some attempts in IR that learn sparse representations of documents for duplicate detection (Hajishirzi et al., 2010), inverted indexing (Zamani et al., 2018), and more. Unlike previous works, however, our method does not require hand-eng”
Overview of the TREC 2019 deep learning track (2020, methodology): “However, we hope that as the body of literature on neural methods for phase 1 retrieval ( e.g. , [Boytsov et al., 2016, Zamani et al., 2018, Mitra et al., 2019, Nogueira et al., 2019]) grows, we would see a larger number of runs with deep learning as an ingredient for phase 1 in future editions of…”