Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation

By Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, Allan Hanbury. 2020

290 citations recorded by the source corpus.

The latency of neural ranking models at query time is largely dependent on the architecture and deliberate choices by their designers to trade-off effectiveness for higher efficiency. This focus on low query latency of a rising number of efficient ranking architectures make them feasible for production deployment. In machine learning an increasingly common approach to close the effectiveness gap of more efficient models is to apply knowledge distillation from a large teacher model to a smaller student model. We find that different ranking architectures tend to produce output scores in different magnitudes. Based on this finding, we propose a cross-architecture training procedure with a margin focused loss (Margin-MSE), that adapts knowledge distillation to the varying score output distributions of different BERT and non-BERT ranking architectures. We apply the teachable information as additional fine-grained labels to existing training triples of the MSMARCO-Passage collection. We evaluate our procedure of distilling knowledge from state-of-the-art concatenated BERT models to four different efficient architectures (TK, ColBERT, PreTT, and a BERT CLS dot product model). We show that across our evaluated architectures our Margin-MSE knowledge distillation significantly improves their effectiveness without compromising their efficiency. To benefit the community, we publish the costly teacher-score training files in a ready-to-use package.

Subfields: Artificial Intelligence, Information Retrieval & Search, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Retrieval & Search

In Efficiency & Compression

Citations in the corpus

Pretrained Transformers for Text Ranking: BERT and Beyond (2020, background): “While there are aspects of text ranking with pretrained trans-formers that are well understood, many promising directions await further exploration.”
Pyserini: An Easy-to-Use Python Toolkit to Support Replicable IR Research with Sparse and Dense Representations (2021, background): “The process of training these encoders can be quite complex, involving, for example, knowledge distillation [10, 16] and complex sampling techniques [26].”
Pyserini: A Python Toolkit for Reproducible Information Retrieval Research with Sparse and Dense Representations (2021, methodology): “In addition to TCT-ColBERT [17], we presently support a number of other models, including DPR [13], ANCE [37], and DistilBERT KD [10].”
Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reduction (2022, methodology): “This is akin to a two stage retrieve and re-rank pipeline [15, 16, 29], but such pipeline have been mostly studied with a separate model per stage (which requires larger indexing resources than our single model).”
TU Wien at TREC DL and Podcast 2021: Simple Compression for Dense Retrieval (2021, methodology): “…task of the podcast track, we apply: 1) Our full TAS-B trained BERT DOT model and re-rank the outcome with our knowledge distilled BERT CAT [5]; 2) Sparse-dense retrieval [12] using our full TAS-B trained model and a standard BM25 approach [18]; and 3) Our full TAS-B trained model, merge…”
ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction (2021, background): “This line of work can be decomposed into three directions: (1) distillation of more expressive architectures (Hofstätter et al., 2020; Lin et al., 2020) including explicit denoising (Qu et al., 2021; Ren et al., 2021b), (2) hard negative sampling (Xiong et al., 2020; Zhan et al., 2020a), and (3)…”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, methodology): “Further, the researchers empirically find that the Margin-MSE loss is more effective than the other options [101], [110], e.”
A Replication Study of Dense Passage Retriever (2021, background): “…an important exemplar of text retrieval using learned dense representations, which has recently emerged as a promising alternative to “traditional” text retrieval using sparse bag-of-words representations (Zhan et al., 2020; Xiong et al., 2020; Hofstätter et al., 2020; Lin et al., 2020).”
SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval (2021, methodology): “While the backbone of these models remains the same, recent works highlight the critical aspects of the training strategy to obtain state-of-the-art results, ranging from improved negative sampling [11, 16] to distillation [12, 16].”
SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking (2021, background): “While the backbone of these models remains the same, recent works highlight the critical aspects of the training strategy to obtain state-of-the-art results, ranging from improved negative sampling [8, 25] to distillation [11, 15].”
BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models (2021, methodology): “Identifying the source for this contrasting behaviour is difficult, as TAS-B and ANCE use different models (DistilBERT vs. RoBERTa-base), a different loss function (InfoNCE [62] vs. Margin-MSE [24] with in-batch negatives), and different hard negative mining strategies.”
A Few Brief Notes on DeepImpact, COIL, and a Conceptual Framework for Information Retrieval Techniques (2021, unknown): “…retrieval, exempliﬁed by DPR (Karpukhin et al., 2020) and ANCE (Xiong et al., 2021), but also encompassing many other techniques (Gao et al., 2021b; Hofstätter et al., 2020; Qu et al., 2021; Hofstätter et al., 2021; Lin Table 1: Our conceptual framework for organizing recent developments in…”
In-Batch Negatives for Knowledge Distillation with Tightly-Coupled Teachers for Dense Retrieval (2021, methodology): “computed due to the computational costs of crossencoders (Hofstätter et al., 2020; Gao et al., 2020a; Barkan et al., 2020).”
Distilling Dense Representations for Ranking using Tightly-Coupled Teachers (2020, methodology): “…models during distillation of dense representations for ranking. inference using the teacher model is interleaved directly during the distillation process: This is a key difference between our approach and previous methods where query–document scores are precomputed (Hofst¨atter et al., 2020).”
Boosted Dense Retriever (2021, methodology): “Dense, MIPS-based Retrieval using bi-encoder architectures leveraging contrastive training with gold pairs (Yih et al., 2011) has recently shown to be effective in several settings (Lee et al., 2019; Karpukhin et al., 2020; Reimers and Gurevych, 2019; Hofst¨atter et al., 2021b).”