Understanding BERT Rankers Under Distillation

By Luyu Gao, Zhuyun Dai, Jamie Callan. 2020

58 citations recorded by the source corpus.

Deep language models, such as BERT pre-trained on large corpora, have given a huge performance boost to state-of-the-art information retrieval ranking systems. Knowledge embedded in such models allows them to pick up complex matching signals between passages and queries. However, the high computation cost during inference limits their deployment in real-world search scenarios. In this paper, we study if and how the knowledge for search within BERT can be transferred to a smaller ranker through distillation. Our experiments demonstrate that it is crucial to use a proper distillation procedure, which produces up to nine times speedup while preserving the state-of-the-art performance.

Subfields: Artificial Intelligence, Management Science and Operations Research, Information Retrieval & Search, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Information Retrieval & Search

In Natural Language Processing

Citations in the corpus

Pretrained Transformers for Text Ranking: BERT and Beyond (2020, background): “One key limitation of BERT is its inability to handle long input sequences and hence difficulty in ranking texts beyond a certain length (e.g., “full-length” documents such as news articles).”
Boosted Dense Retriever (2021, background): “Distillation for dense retrievers is an active area, and more complex schemes exist which could improve results further (Izacard and Grave, 2021; Qu et al., 2021; Yang and Seo, 2020; Lin et al., 2021; Hofst¨atter et al., 2021a; Barkan et al., 2020; Gao et al., 2020).”
In-Batch Negatives for Knowledge Distillation with Tightly-Coupled Teachers for Dense Retrieval (2021, background): “A cross-encoder has been shown to be an effective teacher (Hofstätter et al., 2020; Gao et al., 2020a) since it allows rich interactions between the intermediate transformer representations of a query q and a passage p .”
Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling (2021, methodology): “Apart from specifically training dense retrieval models, knowledge distillation gains popularity, with general-purpose BERT-style models [17, 33] as well as a range of applications in IR: from sequential recommendation models [35], BERT-based retrieval chatbots [36], BERT-based Question Answering [1”
Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation (2020, methodology): “KD has been applied in the ranking task for the same architecture with fewer layers [5, 13, 24] and in constrained sub-tasks, such as keyword-list matching [25].”