Investigating the Successes and Failures of BERT for Passage Re-Ranking

By Harshith Padigela, Hamed Zamani, W. Bruce Croft. 2019

47 citations recorded by the source corpus.

The bidirectional encoder representations from transformers (BERT) model has recently advanced the state-of-the-art in passage re-ranking. In this paper, we analyze the results produced by a fine-tuned BERT model to better understand the reasons behind such substantial improvements. To this aim, we focus on the MS MARCO passage re-ranking dataset and provide potential reasons for the successes and failures of BERT for retrieval. In more detail, we empirically study a set of hypotheses and provide additional analysis to explain the successful performance of BERT.

Subfields: Artificial Intelligence, Management Science and Operations Research, Information Retrieval & Search, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Information Retrieval & Search

In Evaluation & Benchmarks

Citations in the corpus

Pretrained Transformers for Text Ranking: BERT and Beyond (2020, background): “ offer limited insight on how notions of relevance are captured by BERT. Information retrieval researchers, however, have attempted to speciﬁcally examine relevance matching by BERT in ranking tasks [Padigela et al., 2019, Qiao et al., 2019, Zhan et al., 2020a]. While progress has been made, it is f”
On Identifiability in Transformers (2019, background): “…2017; Vig, 2019; Marecek & Rosa, 2018; Clark et al., 2019; Raganato & Tiedemann, 2018; Voita et al., 2019; Tang et al., 2018; Wangperawong, 2018; Padigela et al., 2019; Baan et al., 2019; Lin et al., 2019; Dehghani et al., 2019; Zenkel et al., 2019; Klein & Nabi, 2019; Coenen et al., 2019) may…”
Multi-step Entity-centric Information Retrieval for Multi-Hop Question Answering (2019, positive): “Bag-of-words and contextual embedding models, such as word2vec and BERT, have also been explored extensively for various IR tasks, from document to sentence-level retrieval (Padigela et al., 2019; Zamani and Croft, 2016, 2017).”
Cross-Domain Modeling of Sentence-Level Evidence for Document Retrieval (2019, unknown): “(2019) and unrefereed manuscripts (Qiao et al., 2019; Padigela et al., 2019).”
Asking Clarifying Questions in Open-Domain Information-Seeking Conversations (2019, unknown): “BERT has recently outperformed state-of-the-art models in a number of language understanding and retrieval tasks [17, 27].”
Local Self-Attention over Long Text for Efficient Document Retrieval (2020, background): “More recently, Transformer [7] based architectures have been employed to learn contextual representations which have led to bigger improvements [11, 18, 19, 24].”
The neural hype, justified! (2019, background): “Over the past year, a number of studies [23, 5, 11, 16, 14, 25] have independently established this empirical result.”
Selective Weak Supervision for Neural Information Retrieval (2020, background): “The strong effectiveness, especially considering BERT is also pre-trained on word co-occurrence signals, has raised many investigations of its source of effectiveness in ranking [4, 31, 37].”
MarkedBERT: Integrating Traditional IR Cues in Pre-trained Language Models for Passage Retrieval (2020, methodology): “In order to understand the performance improvement of MarkedBERT across different query types, we follow the comparison method used in [11].”
Information Retrieval: A Biomedical and Health Perspective (2020, background): “Research Network (PCORNet25) [215] and the Observational Health Data Sciences and Informatics (OHDSI26) [216] initiatives.”
Guided Transformer: Leveraging Multiple External Sources for Representation Learning in Conversational Search (2020, methodology): “d on Transformer which is pre-trained for a language modeling task. BERT has recently proven to be effective in a wide range of NLP and IR tasks, including question answering [18], passage re-ranking [32, 34], query performance prediction [20], and conversational QA [40]. The coloring in Figure 2 sh”
Cross-Domain Sentence Modeling for Relevance Transfer with BERT (2019, background): “[48] prioritize studying the reasons behind the gains that come with re-ranking MS MARCO passages with BERT.”
BERT-QE: Contextualized Query Expansion for Document Re-ranking (2020, methodology): “Additionally, the behaviour of BERT when applied to the re-ranking tasks is investigated in (Qiao et al., 2019; Padigela et al., 2019).”