In-Batch Negatives for Knowledge Distillation with Tightly-Coupled Teachers for Dense Retrieval

By Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy J. Lin. 2021

221 citations recorded by the source corpus.

We present an efficient training approach to text retrieval with dense representations that applies knowledge distillation using the ColBERT late-interaction ranking model. Specifically, we propose to transfer the knowledge from a bi-encoder teacher to a student by distilling knowledge from ColBERT’s expressive MaxSim operator into a simple dot product. The advantage of the bi-encoder teacher–student setup is that we can efficiently add in-batch negatives during knowledge distillation, enabling richer interactions between teacher and student models. In addition, using ColBERT as the teacher reduces training cost compared to a full cross-encoder. Experiments on the MS MARCO passage and document ranking tasks and data from the TREC 2019 Deep Learning Track demonstrate that our approach helps models learn robust representations for dense retrieval effectively and efficiently.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

Citations in the corpus

You Only Need One Model for Open-domain Question Answering (2021, methodology): “Per-token embeddings or multiple embeddings were used in ColBERT (Khattab et al., 2020), ME-BERT (Luan et al., 2021), Lin et al. (2021), Lee et al. (2021).”
Promptagator: Few-shot Dense Retrieval From 8 Examples (2022, methodology): “Distilling interaction based models into representation based models has been shown effective in closing the gap between the two (Hofstätter et al., 2020; Ren et al., 2021a; Lin et al., 2021; Ren et al., 2021b; Reddi et al., 2021; Zhang et al., 2022).”
Precise Zero-Shot Dense Retrieval without Relevance Labels (2022, unknown): “A variety of meth-ods such as negative mining (Xiong et al., 2021; Qu et al., 2021), distillation (Qu et al., 2021; Lin et al., 2021b; Hofstätter et al., 2021) and task-specific pre-training (Izacard et al., 2021; Gao and Callan, 2021; Lu et al., 2021; Gao and Callan, 2022; Liu and Shao, 2022) have…”
A Few Brief Notes on DeepImpact, COIL, and a Conceptual Framework for Information Retrieval Techniques (2021, unknown): “Lin et al. (2021) reported the results of dense–sparse hybrids when TCT-ColBERTv2 TCT-ColBERTv2, row ( 3f) , With these hybrid combinations, we are able to achieve, to our knowledge, the highest reported scores on the MS MARCO passage ranking task for single-stage techniques (i.e., no reranking).”
Boosted Dense Retriever (2021, background): “The success of Dense Retrieval has led to many recent papers proposing schemes to improve dense retriever training by in-novating on how negatives are sampled (Xiong et al., 2020; Qu et al., 2021; Zhan et al., 2021c; Lin et al., 2021, inter alia.)”
Dialog Inpainting: Turning Documents into Dialogs (2022, methodology): “Similar to Yu et al. (2021) and Lin et al. (2021), we use hard negatives mined from a base retriever during fine-tuning (See Appendix C.3 for details).”
Mr. TyDi: A Multi-lingual Benchmark for Dense Retrieval (2021, methodology): “Well-known examples include DPR (Karpukhin et al., 2020), ANCE (Xiong et al., 2021), and ColBERT (Khattab and Zaharia, 2020), but there is much recent work along these lines (Gao et al., 2020; Hofstätter et al., 2020; Hof-stätter et al., 2021; Lin et al., 2021b), just to list a few papers.”