Multi-Stage Document Ranking with BERT

By Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, Jimmy J. Lin. 2019

502 citations recorded by the source corpus.

The advent of deep neural networks pre-trained via language modeling tasks has spurred a number of successful applications in natural language processing. This work explores one such popular model, BERT, in the context of document ranking. We propose two variants, called monoBERT and duoBERT, that formulate the ranking problem as pointwise and pairwise classification, respectively. These two models are arranged in a multi-stage ranking architecture to form an end-to-end search system. One major advantage of this design is the ability to trade off quality against latency by controlling the admission of candidates into each pipeline stage, and by doing so, we are able to find operating points that offer a good balance between these two competing metrics. On two large-scale datasets, MS MARCO and TREC CAR, experiments show that our model produces results that are either at or comparable to the state of the art. Ablation studies show the contributions of each component and characterize the latency/quality tradeoff space.

Subfields: Artificial Intelligence, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Retrieval & Search

Citations in the corpus

Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval (2020, methodology): “The reranking step often applies BERT on top of the sparse-retrieved documents, i.e. by concatenating them with the query and feeding into a ﬁne-tuned BERT reranker [1, 19].”
ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT (2020, methodology): “Subsequent work by Nogueira et al. [27] introduced duoBERT, which fine-tunes BERT to compare the relevance of a pair of documents given a query.”
R2-D2: A Modular Baseline for Open-Domain Question Answering (2021, background): “For document ranking, Nogueira et al. (2019) proposed a multistage architecture.”
Sparse, Dense, and Attentional Representations for Text Retrieval (2020, background): “This two stage approach is powerful and has achieved state-of-the-art results on multiple IR benchmarks (Nogueira and Cho, 2019; Yang et al., 2019; Nogueira et al., 2019a), especially since largescale annotated data has become available for training deep neural models (Dietz et al., 2018; Craswell…”
Document Ranking with a Pretrained Sequence-to-Sequence Model (2020, methodology): “Main results on the MS MARCO passage retrieval task are shown in Table 1, comparing BERTlarge (Nogueira et al., 2019a) to T5 models of different sizes.”
Pyserini: An Easy-to-Use Python Toolkit to Support Replicable IR Research with Sparse and Dense Representations (2021, background): “957 Pyserini: multi-stage pipelines (4d) monoBERT [20] 0.”
Pyserini: A Python Toolkit for Reproducible Information Retrieval Research with Sparse and Dense Representations (2021, methodology): “957 Pyserini: multi-stage pipelines (4d) monoBERT [27] 0.”
RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering (2020, methodology): “Recently, researchers have utilized deep learning to improve traditional passage retrievers, including document expansions (Nogueira et al., 2019c), question expansions (Mao et al., 2020) and term weight estimation (Dai and Callan, 2019).”
Learning Passage Impacts for Inverted Indexes (2021, methodology): “In particular, Table 4 shows that the gap with the other methods is greater with smaller cutoff values, which reduces the re-ranking cost and thus could enable the use of more complex pairwise ranking models, such as DuoBERT [20].”
PROP: Pre-training with Representative Words Prediction for Ad-hoc Retrieval (2020, methodology): “, ELMo [27] and BERT, in the context of ad-hoc document ranking, and showed that they can also largely benefit the search tasks where training data are limited [10, 21, 24, 25, 40].”
Shallow pooling for sparse labels (2021, background): “[29] as provided by their github repository.”
From doc2query to docTTTTTquery (2019, unknown): “Note that in a multi-stage reranking architecture, this represents the initial candidate generation stage; other techniques, such as mono/duoBERT [6], can be applied to further improve effectiveness.”
Large Language Models for Information Retrieval: A Survey (2023, unknown): “Enc-only [140] Enc-dec [13], [141], [142], [143] Dec-only [131], [144], [145] Pointwise [146], [147], [148], [149], [150], [151] Listwise [152], [153], [154] Pairwise [155], [156] Data Augmentation - [157], [158], [159], [160], [161], [162] with lower inference time demands), the potential mismatch…”
MarkedBERT: Integrating Traditional IR Cues in Pre-trained Language Models for Passage Retrieval (2020, methodology): “The same trend is witnessed in the IR community, where several works emerged applying these models to IR tasks such as document retrieval [1, 5, 9], passage ranking [8], and question answering [17].”
SGPT: GPT Sentence Embeddings for Semantic Search (2022, unknown): “To balance the trade-offs, multi-stage architectures have been proposed [30, 34, 21].”
Long Document Ranking with Query-Directed Sparse Transformer (2020, background): “And recently the state-of-the-art in many text ranking tasks has been taken by BERT or other pretrained language models (Devlin et al., 2019; Nogueira et al., 2019; Nogueira and Cho, 2019; Dai and Callan, 2019; Yang et al., 2019; Craswell et al., 2020), when sufficient relevance labels are available”
RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking (2021, methodology): “…re-triever BM25 (Yang et al., 2017) and ﬁve traditional retrievers enhanced by neural networks, including doc2query (Nogueira et al., 2019c), DeepCT (Dai and Callan, 2019), docTTTTT-query (Nogueira et al., 2019a), GAR (Mao et al., 2020), UHD-BERT (Jang et al., 2021) and COIL (Gao et al., 2021a).”
BISON: BM25-weighted Self-Attention Framework for Multi-Fields Document Search (2020, background): “Typically, Nogueira et al.(22) built a multi-stage ranking architecture on BERT by formulating the ranking problem as pointwise and pairwise classiﬁcation, respectively.”
Text and Code Embeddings by Contrastive Pre-Training (2022, methodology): “It also outperforms docT5query (Nogueira et al., 2019a) that relies on a fine-tuned T5 model (Raffel et al., 2019) for document expansion. cpt-text results are competitive even with methods that use substantially more compute at test time.”