Distilling Dense Representations for Ranking using Tightly-Coupled Teachers

By Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy J. Lin. 2020

144 citations recorded by the source corpus.

We present an approach to ranking with dense representations that applies knowledge distillation to improve the recently proposed late-interaction ColBERT model. Specifically, we distill the knowledge from ColBERT's expressive MaxSim operator for computing relevance scores into a simple dot product, thus enabling single-step ANN search. Our key insight is that during distillation, tight coupling between the teacher model and the student model enables more flexible distillation strategies and yields better learned representations. We empirically show that our approach improves query latency and greatly reduces the onerous storage requirements of ColBERT, while only making modest sacrifices in terms of effectiveness. By combining our dense representations with sparse representations derived from document expansion, we are able to approach the effectiveness of a standard cross-encoder reranker using BERT that is orders of magnitude slower.

Subfields: Artificial Intelligence, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

Pyserini: An Easy-to-Use Python Toolkit to Support Replicable IR Research with Sparse and Dense Representations (2021, methodology): “Here, we are using TCT-ColBERT [16], a model our group has constructed from ColBERT [13] using knowledge distillation.”
Pyserini: A Python Toolkit for Reproducible Information Retrieval Research with Sparse and Dense Representations (2021, methodology): “For dense retrieval, results using TCT-ColBERT [17] are shown in rows (2) using different indexes.”
TU Wien at TREC DL and Podcast 2021: Simple Compression for Dense Retrieval (2021, methodology): “…we apply: 1) Our full TAS-B trained BERT DOT model and re-rank the outcome with our knowledge distilled BERT CAT [5]; 2) Sparse-dense retrieval [12] using our full TAS-B trained model and a standard BM25 approach [18]; and 3) Our full TAS-B trained model, merge the outcome with BM25 rankings…”
Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling (2021, methodology): “[23] showed the benefit of utilizing a ColBERT teacher model for in-batch negative signals.”
Jointly Optimizing Query Encoder and Product Quantization to Improve Retrieval Performance (2021, methodology): “The baseline trained by knowledge distillation is TCT-ColBERT [30].”
Overview of Touché 2022: Argument Retrieval (2022, unknown): “For Task 2, in terms of relevance, a TCT-ColBERT ranker [9] and, in terms of quality, a combination of query-dependent BM25F scores [10] and predicted argument quality were most effective.”
Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reduction (2022, methodology): “This is akin to a two stage retrieve and re-rank pipeline [15, 16, 29], but such pipeline have been mostly studied with a separate model per stage (which requires larger indexing resources than our single model).”
ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction (2021, background): “This line of work can be decomposed into three directions: (1) distillation of more expressive architectures (Hofstätter et al., 2020; Lin et al., 2020) including explicit denoising (Qu et al., 2021; Ren et al., 2021b), (2) hard negative sampling (Xiong et al., 2020; Zhan et al., 2020a), and (3)…”
A Replication Study of Dense Passage Retriever (2021, background): “…an important exemplar of text retrieval using learned dense representations, which has recently emerged as a promising alternative to “traditional” text retrieval using sparse bag-of-words representations (Zhan et al., 2020; Xiong et al., 2020; Hofstätter et al., 2020; Lin et al., 2020).”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, methodology): “training 2020 AugmentSBERT [102] BERTbase SR NLL SHN+DeHN KD Kernel density estimation 2020 TCT-ColBERT [103] BERTbase SR KL IB+SHN KD Sparse-dense hybrid 2020 Multi-stage [104] BERTbase SR NLL IB+SHN KD TAP Embedding fusion 2020 DKRR [105] BERTbase SR KL SHN KD Iter.”
SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking (2021, methodology): “We compare to the following sparse approaches (1) BM25 (2) DeepCT [4] (3) doc2query-T5 [18] (4) and SparTerm [1], as well as state-of-the-art dense approaches ANCE [25] and TCT-ColBERT [15].”
Bridging the Gap Between Indexing and Retrieval for Differentiable Search Index with Query Generation (2022, methodology): “In our cross-encoder ranker, all the query tokens can interact with all the document tokens thus it has more substantial relevance modeling power than other ranker architectures, such as dual- or bi-encoders [29, 31, 40, 53].”
Rethinking Search: Making Experts out of Dilettantes (2021, background): “resulted in a shift away from traditional inverted indexes towards dense vector-based indexes (or hybrid inverted + vector-based indexes) [28, 40, 41, 44, 45, 49, 102].”
Optimizing Dense Retrieval Model Training with Hard Negatives (2021, methodology): “Since TCT-ColBERT [18] is not open-sourced, we borrow the results from its original paper and do not perform the significance test on it.”