Learning Passage Impacts for Inverted Indexes

By Antonio Mallia, Omar Khattab, Nicola Tonellotto, Torsten Suel. 2021

197 citations recorded by the source corpus.

Neural information retrieval systems typically use a cascading pipeline, in which a first-stage model retrieves a candidate set of documents and one or more subsequent stages re-rank this set using contextualized language models such as BERT. In this paper, we propose DeepImpact, a new document term-weighting scheme suitable for efficient retrieval using a standard inverted index. Compared to existing methods, DeepImpact improves impact-score modeling and tackles the vocabulary-mismatch problem. In particular, DeepImpact leverages DocT5Query to enrich the document collection and, using a contextualized language model, directly estimates the semantic importance of tokens in a document, producing a single-value representation for each token in each document. Our experiments show that DeepImpact significantly outperforms prior first-stage retrieval approaches by up to 17% on effectiveness metrics w.r.t. DocT5Query, and, when deployed in a re-ranking scenario, can reach the same effectiveness of state-of-the-art approaches with up to 5.1x speedup in efficiency.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

Citations in the corpus

Autoregressive Search Engines: Generating Substrings as Document Identifiers (2022, background): “…on term-weighting schemes, with many recent works proposing sophisticated, contextualized weights for both queries, and documents(Dai and Callan, 2019; Gao et al., 2021; Lin and Ma, 2021; Mallia et al., 2021; Dai and Callan, 2020; Bai et al., 2020; Zhao et al., 2021; Formal et al., 2021b,a).”
SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval (2021, background): “Recently, DeepImpact [18] combined the expansion from doc2queryT5 with the re-weighting from DeepCT to learn term impacts.”
Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reduction (2022, background): “There are concurrent efforts revisiting lexical matching with learned sparse representations [10, 12, 26] or learned passage impacts [37], which employ the efficiency of exact lexical matches.”
Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One? (2021, methodology): “Various methods have been proposed to improve term weight learning (Dai and Callan, 2020; Mallia et al., 2021), address vocabulary mismatch (Nogueira and Lin, 2019) and se-”
A Few Brief Notes on DeepImpact, COIL, and a Conceptual Framework for Information Retrieval Techniques (2021, unknown): “This ablation experiment was not reported in Mallia et al. (2021), but would be interesting to conduct.”
Bridging the Gap Between Indexing and Retrieval for Differentiable Search Index with Query Generation (2022, methodology): “Most modern information retrieval systems are based on the index-then-retrieve pipeline where documents are first encoded and stored in an inverted index [10, 13, 15, 26, 34, 55, 58] or a nearest neighbor search index [7, 14, 20, 21, 25, 29, 49, 51] and search results are then constructed based on a”
ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction (2021, background): “single dimension, reducing them to scalar weights that extend models like DeepCT (Dai and Callan, 2020) and DeepImpact (Mallia et al., 2021).”
Latent Terms: Dense Retrievers Contain Trivially Extractable BM25-ready Zipfian Vocabularies (2026, unknown): “Initially, work focused on the document-side: DeepCT (Dai and Callan, 2019) and DeepImpact (Mallia et al., 2021) developed contextualized methods to learn sparse document representations, while Doc2Query approaches (Gospodinov et al., 2023; Nogueira et al., 2019) focused on vocabulary expansion to…”
A Survey of Model Architectures in Information Retrieval (2025, unknown): “At a higher level, LSR can be viewed as a way to learn token importance or “impact” scores from data (Dai and Callan, 2019b; Bai et al., 2020; Mallia et al., 2021), in contrast to static formulas like BM25.”