Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrieval

By Zhuyun Dai, Jamie Callan. 2019

219 citations recorded by the source corpus.

Term frequency is a common method for identifying the importance of a term in a query or document. But it is a weak signal, especially when the frequency distribution is flat, such as in long queries or short documents where the text is of sentence/passage-length. This paper proposes a Deep Contextualized Term Weighting framework that learns to map BERT's contextualized text representations to context-aware term weights for sentences and passages. When applied to passages, DeepCT-Index produces term weights that can be stored in an ordinary inverted index for passage retrieval. When applied to query text, DeepCT-Query generates a weighted bag-of-words query. Both types of term weight can be used directly by typical first-stage retrieval algorithms. This is novel because most deep neural network based ranking models have higher computational costs, and thus are restricted to later-stage rankers. Experiments on four datasets demonstrate that DeepCT's deep contextualized text understanding greatly improves the accuracy of first-stage retrieval algorithms.

Subfields: Artificial Intelligence, Information Systems, Information Retrieval & Search, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Information Retrieval & Search

In Natural Language Processing

Citations in the corpus

Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval (2020, background): “Addressing this vocabulary mismatch is a core research topic in IR [8, 20, 21, 22].”
ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT (2020, methodology): “For TREC CAR, we follow related work [3, 25] and use a different pre-trained model to the official ones.”
Autoregressive Search Engines: Generating Substrings as Document Identifiers (2022, background): “…depends heavily on term-weighting schemes, with many recent works proposing sophisticated, contextualized weights for both queries, and documents(Dai and Callan, 2019; Gao et al., 2021; Lin and Ma, 2021; Mallia et al., 2021; Dai and Callan, 2020; Bai et al., 2020; Zhao et al., 2021; Formal et…”
Sparse, Dense, and Attentional Representations for Text Retrieval (2020, methodology): “3) DEEPCT is the DeepCT-Index model introduced in Dai and Callan (2020) which learns to map BERTâĂŹs contextualized text representations to contextaware term weights for sentences and passages.”
A Few Brief Notes on DeepImpact, COIL, and a Conceptual Framework for Information Retrieval Techniques (2021, unknown): “This leads us to the next important observation: Relating DeepCT, DeepImpact, and COIL.”
Modularized Transfomer-based Ranking Framework (2020, methodology): “To train MORES, we connect the three Transformers and enforce module coupling with end-to-end training using the pointwise loss function (Dai and Callan, 2019).”
SLEDGE: A Simple Yet Effective Baseline for Coronavirus Scientific Knowledge Search (2020, background): “Others later observed effectiveness at document ranking tasks, showing that these models can handle natural-language questions better than prior approaches (Dai and Callan, 2019b) and that they can be incorporated into prior neutral ranking techniques (MacAvaney et al., 2019).”
SparTerm: Learning Term-based Sparse Representation for Fast Text Retrieval (2020, methodology): “• DeepCT [2] is a deep contextualized term weighting model which maps the BERT’s representations to term weightings for retrieval.”
SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval (2021, background): “Recently,DeepImpact[18]combinedtheexpansionfromdoc2query-T5 with the re-weighting from DeepCT to learn term impacts .”
RepBERT: Contextualized Text Embeddings for First-Stage Retrieval (2020, methodology): “Although some recent works [1, 2, 3] adopt deep language models to improve bag-of-words approaches, they still rely on exact term match signals and can hardly retrieve documents on semantic level.”
EARL: Speedup Transformer-based Rankers with Pre-computed Representation (2020, background): “As a result, transformer-based rankers that take long to score a single query-document pair needs to adopt a shorter candidate list to rerank, which may lead to substantial drops in final reranking effectiveness [5].”
Complementing Lexical Retrieval with Semantic Residual Embedding (2020, methodology): “• DeepCT : DeepCT (Dai and Callan, 2019a) is a state-of-the-art deep lexical retrieval model that uses BERT (Devlin et al., 2018) to estimate the semantic importance of words in a document.”
COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted List (2021, methodology): “We also report recall for the dev queries following prior work (Dai and Callan, 2019a; Nogueira and Lin, 2019).”
Jointly Optimizing Query Encoder and Product Quantization to Improve Retrieval Performance (2021, methodology): “• DeepCT [13]: It re-weights termweights for BM25with BERT [15].”
Expansion via Prediction of Importance with Contextualization (2020, methodology): “This also outperforms the DeepCT performance reported by [2] of 0.”