Passage Re-ranking with BERT

By Rodrigo Nogueira, Kyunghyun Cho. 2019

1,491 citations recorded by the source corpus.

Recently, neural models pretrained on a language modeling task, such as ELMo (Peters et al., 2017), OpenAI GPT (Radford et al., 2018), and BERT (Devlin et al., 2018), have achieved impressive results on various natural language processing tasks such as question-answering and natural language inference. In this paper, we describe a simple re-implementation of BERT for query-based passage re-ranking. Our system is the state of the art on the TREC-CAR dataset and the top entry in the leaderboard of the MS MARCO passage retrieval task, outperforming the previous state of the art by 27% (relative) in MRR@10. The code to reproduce our results is available at this https URL

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

In Large Language Models

Citations in the corpus

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, background): “Learned Retrieval There is signiﬁcant work on learning to retrieve documents in information retrieval, more recently with pre-trained, neural language models [39, 22] similar to ours.”
Dense Passage Retrieval for Open-Domain Question Answering (2020, background): “rse one. While not the focus of this work, dense representations from pretrained models, along with cross-attention mechanisms, have also been shown effective in passage or dialogue re-ranking tasks (Nogueira and Cho, 2019;Humeau et al.,2020). Finally, a concurrent work (Khattab and Zaharia,2020) de”
Distilling Knowledge from Reader to Retriever for Question Answering (2020, methodology): “This lead Nogueira & Cho (2019) and Yang et al. (2019) to use a BERT model to jointly embed documents and query, a technique known as cross-encoder.”
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval (2020, methodology): “For example, search ranking [1], open domain question answering [2], and fact veriﬁcation [3, 4] all ﬁrst retrieve relevant documents as the input to their later stage reranking, machine reading, and reasoning models.”
ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT (2020, methodology): “Similar to [25], we fine-tune all ColBERT models with learning rate 3 × 10 − 6 with a batch size 32.”
Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering (2019, background): “Re-rank (Nogueira & Cho, 2019) that learns to retrieve paragraphs by ﬁne-tuning BERT to re-rank the top F TF-IDF paragraphs.”
Re2G: Retrieve, Rerank, Generate (2022, methodology): “The reranker we use is based on the sequence-pair classiﬁcation of Nogueira and Cho [2019].”
Pretrained Transformers for Text Ranking: BERT and Beyond (2020, methodology): “359 (2c) + monoBERTBase [Nogueira and Cho, 2019] 0.”
BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model (2019, methodology): “Our experiments demonstrate the potential of us-ing BERT as a standalone language model rather than as a parameter initializer for transfer learning (Devlin et al., 2018; Lample and Conneau, 2019; Nogueira and Cho, 2019).”
Sparse, Dense, and Attentional Representations for Text Retrieval (2020, methodology): “Due to the computational cost, crossattentional BERT is applied only in reranking as in prior work (Nogueira and Cho, 2019; Yang et al., 2019).”
Masked Language Model Scoring (2019, methodology): “Contemporary examples are the use of finetuned BERT scores in a question-answering pipeline (Nogueira and Cho, 2019), or “as-is” cosine similarity scores from BERT to evaluate generated text (Zhang et al.”
Document Ranking with a Pretrained Sequence-to-Sequence Model (2020, methodology): “Deep transformer models pretrained with language modeling objectives, exempliﬁed by BERT [3], have proven highly effective in a variety of classiﬁcation and sequence labeling tasks in NLP. Nogueira and Cho [9] were the ﬁrst to demonstrate its effectiveness in ranking tasks.”
Blockwise Self-Attention for Long Document Understanding (2019, background): “, 2019) and text retrieval (Lee et al., 2019; Nogueira and Cho, 2019), to name a few.”
Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrieval (2019, methodology): “The experimental settings mainly follow the methodology used in previous neural passage ranking/re-ranking studies [22, 23] MS MARCO [21] is a question-to-passage retrieval dataset with 8 .”
Multi-Stage Document Ranking with BERT (2019, methodology): “Our work makes the following contributions: We start by describing monoBERT, a pointwise classiﬁcation model of document relevance that was introduced in Nogueira and Cho (2019).”
Document Expansion by Query Prediction (2019, methodology): “We optionally re-rank these retrieved documents using BERT (Devlin et al., 2018) as described in Nogueira and Cho (2019).”
Simple Applications of BERT for Ad Hoc Document Retrieval (2019, methodology): “Following Nogueira and Cho (2019), BERT is used for binary classiﬁcation (i.e., relevance) by taking the [CLS] vector as input to a single layer neural network.”