Understanding the Behaviors of BERT in Ranking

By Yifan Qiao, Chenyan Xiong, Zhenghao Liu, Zhiyuan Liu. 2019

233 citations recorded by the source corpus.

This paper studies the performances and behaviors of BERT in ranking tasks. We explore several different ways to leverage the pre-trained BERT and fine-tune it on two ranking tasks: MS MARCO passage reranking and TREC Web Track ad hoc document ranking. Experimental results on MS MARCO demonstrate the strong effectiveness of BERT in question-answering focused passage ranking tasks, as well as the fact that BERT is a strong interaction-based seq2seq matching model. Experimental results on TREC show the gaps between the BERT pre-trained on surrounding contexts and the needs of ad hoc document ranking. Analyses illustrate how BERT allocates its attentions between query-document tokens in its Transformer layers, how it prefers semantic matches between paraphrase tokens, and how that differs with the soft match patterns learned by a click-trained neural ranker.

Subfields: Artificial Intelligence, Information Systems, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Large Language Models

In Natural Language Processing

Citations in the corpus

Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval (2020, background): “This overthrows a previously-held belief that it is necessary to capture the interactions between the discrete query and document terms [34, 35].”
ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT (2020, background): “By computing deeply-contextualized semantic representations of query–document pairs, these LMs help bridge the pervasive vocabulary mismatch [22, 42] between documents and queries [30].”
R2-D2: A Modular Baseline for Open-Domain Question Answering (2021, methodology): “Over time, Bi-LSTM were replaced by BERT-like transformer encoders (Qiao et al., 2019; Wang et al., 2019a).”
Pretrained Transformers for Text Ranking: BERT and Beyond (2020, background): “on how notions of relevance are captured by BERT. Information retrieval researchers, however, have attempted to speciﬁcally examine relevance matching by BERT in ranking tasks [Padigela et al., 2019, Qiao et al., 2019, Zhan et al., 2020a]. While progress has been made, it is fair to say that we stil”
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (2019, methodology): “…this limitations, researchers passed single sentences through BERT and then derive a ﬁxed sized vector by either averaging the outputs (similar to average word embeddings) or by using the output of the special CLS token (for example: May et al. (2019); Zhang et al. (2019); Qiao et al. (2019)).”
Context-Aware Document Term Weighting for Ad-Hoc Search (2020, methodology): “Although much recent work has applied contextualized neural language models like BERT [14] to IR tasks, prior work mainly focused on using BERT as black-box re-ranker [9, 27, 28, 31].”
BERT-QE: Contextualized Query Expansion for Document Re-ranking (2020, methodology): “Additionally, the behaviour of BERT when applied to the re-ranking tasks is investigated in (Qiao et al., 2019; Padigela et al., 2019).”
Reader-Guided Passage Reranking for Open-Domain Question Answering (2021, background): “An alternative solution is to rerank the initial retrieval results via a reranker, which is widely used in information retrieval (Nogueira and Cho, 2019; Qiao et al., 2019) and explored in early OpenQA systems (Wang et al.”
Rethinking Query Expansion for BERT Reranking (2020, result): “This behaviour has been previously studied in previous work [15] wherein BERT favors text sequence pairs that are close in semantic meaning.”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, methodology): “To implement the reranker, a typical approach is to employ the cross-encoder as the ranking model, showing substantial improvements over the traditional methods [193], [272]–[274].”
Knowledge-Grounded Dialogue Generation with Pre-trained Language Models (2020, background): “…2019; Lewis et al., 2019), and therefore are revolutionizing almost the full spectrum of NLP applications (Raffel et al., 2019; Sun et al., 2019b; Qiao et al., 2019; Zhang et al., 2019b; Lam-ple and Conneau, 2019) and some interdisciplinary applications in NLP and computer vision (Lu et al.,…”
Selective Weak Supervision for Neural Information Retrieval (2020, methodology): “The strong effectiveness, especially considering BERT is also pre-trained on word co-occurrence signals, has raised many investigations of its source of effectiveness in ranking [4, 31, 37].”
Cross-Domain Sentence Modeling for Relevance Transfer with BERT (2019, methodology): “[51] also examines the performance and behavior of BERT when used as a re-ranker for passage ranking on MS MARCO and for document ranking on the TREC Web Track.”
An Analysis of BERT in Document Ranking (2020, background): “[11] designed a representation-focused [6] BERT ranker.”
Fine-grained Fact Verification with Kernel Graph Attention Network (2019, methodology): “Then we use K kernels to extract the matching feature ~ K(M q→p i ) from the translation matrix M q→p (Xiong et al., 2017; Dai et al., 2018; Qiao et al., 2019; MacAvaney et al., 2019):”
RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking (2021, methodology): “formance (Nogueira and Cho, 2019; Qiao et al., 2019), which can better capture the semantic interactions between the passage and the query, but requires more computational efforts than the dualencoder.”
MarkedBERT: Integrating Traditional IR Cues in Pre-trained Language Models for Passage Retrieval (2020, background): “Furthermore, studies such as [11] and [12] tried to understand the behaviors of BERT that led to its success and failure in passage ranking.”
A Wrong Answer or a Wrong Question? An Intricate Relationship between Question Reformulation and Answer Selection in Conversational Question Answering (2020, background): “Previous work that focused on analysing and explaining performance of Transformer-based models on the ranking task used word attention visualisation and random removals of non-stop words (Dai and Callan, 2019; Qiao et al., 2019).”