CEDR: Contextualized Embeddings for Document Ranking

By Sean MacAvaney, Andrew Yates, Arman Cohan, Nazli Goharian. 2019

371 citations recorded by the source corpus.

Although considerable attention has been given to neural ranking architectures recently, far less attention has been paid to the term representations that are used as input to these models. In this work, we investigate how two pretrained contextualized language models (ELMo and BERT) can be utilized for ad-hoc document ranking. Through experiments on TREC benchmarks, we find that several ex-sting neural ranking architectures can benefit from the additional context provided by contextualized language models. Furthermore, we propose a joint approach that incorporates BERT's classification vector into existing neural models and show that it outperforms state-of-the-art ad-hoc ranking baselines. We call this joint approach CEDR (Contextualized Embeddings for Document Ranking). We also address practical challenges in using these models for ranking, including the maximum input length imposed by BERT and runtime performance impacts of contextualized language models.

Subfields: Artificial Intelligence, Information Retrieval & Search, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Retrieval & Search

In Natural Language Processing

Citations in the corpus

Pretrained Transformers for Text Ranking: BERT and Beyond (2020, methodology): “This limitation is addressed by a number of models [1, 4, 20, 25, 30, 39], and a simple retrieve-then-rerank approach can be elaborated into a multi-stage architecture with reranker pipelines [26, 33, 37] that balance effectiveness and efficiency.”
Document Ranking with a Pretrained Sequence-to-Sequence Model (2020, methodology): “For Robust04 ( adopt a different architecture than CEDR and Birch, but effectiveness clearly improves as the size of the T5 model increases.”
Multi-Stage Document Ranking with BERT (2019, methodology): “One such popular model, BERT, has recently been applied to search-related tasks, retrieval-based question answering (Yang et al., 2019b), as well as document ranking (Yang et al., 2019c; MacAvaney et al., 2019; Yilmaz et al., 2019).”
Efficient Document Re-Ranking for Transformers by Precomputing Term Representations (2020, methodology): “ [47] used BERT in an end-to-end question-answering pipeline. In this setting, they predict the spans of text that answer the question (same setting as demonstrated on SQuAD in [8]). MacAvaney et al. [27] extended that BERT is effective atdocument ranking, both in the “vanilla” setting (learning a r”
Covidex: Neural Ranking Models and Keyword Search Infrastructure for the COVID-19 Open Research Dataset (2020, background): “Despite the success of BERT for document ranking (Dai and Callan, 2019; MacAvaney et al., 2019; Yilmaz et al., 2019), there is evidence that ranking with sequence-to-sequence models can achieve even better effectiveness, particularly in zero-shot and other settings with limited training data…”
Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation (2020, methodology): “The common way of utilizing the BERT pre-trained Transformer model in a re-ranking scenario [29, 34, 45] is by concatenating query and passage input sequences.”
Deep Reinforced Query Reformulation for Information Retrieval (2020, methodology): “In particular, we use DPH, DPH + Bo1 query expansion [1], as well as a BERT re-ranker (as implemented by the CEDR toolkit [25]).”
A Survey of the Usages of Deep Learning for Natural Language Processing (2020, methodology): “[171] extracted query term representations from two pretrained contextualized language models,”
Critically Examining the "Neural Hype": Weak Baselines and the Additivity of Effectiveness Gains from Neural Ranking Models (2019, neutral): “[13] (the authors did not report AP results and hence the paper was excluded from the 18).”
Training Curricula for Open Domain Answer Re-Ranking (2020, methodology): “MacAvaney et al. [23] further demonstrated that signals from contextualized language models can be incorporated into other interaction-focused neural ranking architectures, boosting ranking performance beyond both transformer-based rankers and the non-contextualized interaction models.”
Rethinking Query Expansion for BERT Reranking (2020, methodology): “Other existing BERT rerankers use a similar architecture as BERT-MaxP, and apply more complex techniques to improve accuracy, e.g., domain adaptation [2, 16] , fusion with BM25 [16] , and customized pooling layers [11] .”
Cross-Domain Modeling of Sentence-Level Evidence for Document Retrieval (2019, methodology): “Beyond our own previous work (Yang et al., 2019c), which to our knowledge is the ﬁrst re-ported application of BERT to document retrieval, there have been several other proposed approaches, including MacAvaney et al. (2019) and unrefereed manuscripts (Qiao et al., 2019; Padi-gela et al., 2019).”
MarkedBERT: Integrating Traditional IR Cues in Pre-trained Language Models for Passage Retrieval (2020, background): “The same trend is witnessed in the IR community, where several works emerged applying these models to IR tasks such as document retrieval [1, 5, 9], passage ranking [8], and question answering [17].”
The Cascade Transformer: an Application for Efficient Answer Sentence Selection (2020, methodology): “, 2019), ad-hoc document retrieval (Yang et al., 2019b; MacAvaney et al., 2019), and semantic understanding (Liu et al.”
TU_DBS in the ARQMath Lab 2021, CLEF (2021, background): “The second disadvantage is that the average query latency of BERT-based systems is a few orders of magnitude larger compared to non-neural methods [6, 7], due to the fact that for each query-document pair an entire forward pass through the deep network needs to be performed.”
Capreolus: A Toolkit for End-to-End Neural Ad Hoc Retrieval (2020, result): “However, recent experiments with BERT-based rerankers [1, 9, 19] show impressive improvements on Robust04, even with limited data.”
Information Retrieval: A Biomedical and Health Perspective (2020, background): “Recent US federal research initiatives, such as the AllOfUs (formerly Precision Medicine) Initiative [213] and the 21st Century Cures Act [214], envision an important role for large repositories of data to accompany patients in cutting-edge research.”
SLEDGE: A Simple Yet Effective Baseline for Coronavirus Scientific Knowledge Search (2020, methodology): “This model is akin to the Vanilla BERT ranker from (MacAvaney et al., 2019), but utilizing the SciBERT model base (which is trained on scientiﬁc literature) instead.”