Contrastive Representation Distillation

By Yonglong Tian, Dilip Krishnan, Phillip Isola. 2019

1,372 citations recorded by the source corpus.

Often we wish to transfer representational knowledge from one neural network to another. Examples include distilling a large network into a smaller one, transferring knowledge from one sensory modality to a second, or ensembling a collection of models into a single estimator. Knowledge distillation, the standard approach to these problems, minimizes the KL divergence between the probabilistic outputs of a teacher and student network. We demonstrate that this objective ignores important structural knowledge of the teacher network. This motivates an alternative objective by which we train a student to capture significantly more information in the teacher's representation of the data. We formulate this objective as contrastive learning. Experiments demonstrate that our resulting new objective outperforms knowledge distillation and other cutting-edge distillers on a variety of knowledge transfer tasks, including single model compression, ensemble distillation, and cross-modal transfer. Our method sets a new state-of-the-art in many transfer tasks, and sometimes even outperforms the teacher network when combined with knowledge distillation. Code: this http URL

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Efficiency & Compression

Citations in the corpus

Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need? (2020, background): “Another two related works are [30] which provides theoretical analysis of self-distillation and CRD [50] which shows distillation improves the transferability across datasets.”
Contrastive Multiview Coding (2019, background): “The idea of contrastive learning has also started to spread over many other tasks in various other domains [74, 86, 61, 76, 48, 38, 75].”
Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation (2022, background): “Recently, contrastive representation distillation (Tian et al., 2019) distills the knowledge from the teacher network to the student network by maximizing the mutual information between the two networks, and is recently extended to transfer the knowledge from the pre-”
What makes for good views for contrastive learning (2020, background): “…(d) In the video domain, [64, 44, 39] use a video as v 1 and aligned texts as v 2 , while [47, 10, 53] leverages the correspondence between video and audio; (e) [66] even considers representations from teacher and student networks as v 1 and v 2 , and performs contrastive knowledge distillation.”
Ensemble Distillation for Robust Model Fusion in Federated Learning (2020, background): “By encouraging the student model to approximate the output logits of the teacher model, the student is able to imitate the teacher’s behavior with marginal quality loss (Romero et al., 2015; Zagoruyko & Komodakis, 2016; Kim et al., 2018; Tung & Mori, 2019; Koratana et al., 2019; Huang & Wang, 2017; ”
Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed (2021, background): “, 2015) has been explored extensively in supervised learning tasks such as image classification [26, 46, 44, 7, 36, 43], language modeling [35, 37, 28, 14, 34], and speech recognition [3, 6, 40, 17, 9].”
CLASSIC: Continual and Contrastive Learning of Aspect Sentiment Classification Tasks (2021, methodology): “The intuition of CKS is as follows: Contrasive learning has the ability to capture the shared knowledge between different views (Tian et al., 2020b; van den Oord et al., 2018).”
Divide and Contrast: Self-supervised Learning from Uncurated Data (2021, background): “Finally, contrastive learning has also been applied to supervised image classification [48], image translation [70], knowledge distillation [89, 77], and adversarial learning [49].”
Complementary Relation Contrastive Distillation (2021, methodology): “Following [41, 47], we adopt the ResNet34ResNet18 pair to evaluate the effectiveness of CRCD on ImageNet.”
Hybrid Discriminative-Generative Training via Contrastive Learning (2020, background): “Contrastive learning allows learning good representations and enables efﬁcient training on downstream tasks, an incomplete list includes image classiﬁcation [2, 3, 9, 15, 35], video understanding [13], and knowledge distillation [45].”
Continual Contrastive Self-supervised Learning for Image Classification (2021, methodology): “CRD[46], based on CMC[45], combines self-supervised learning with knowledge distillation.”
Self-supervised Label Augmentation via Input Transformations (2019, neutral): “There are many advanced distillation techniques (Zagoruyko & Komodakis, 2016a; Park et al., 2019; Ahn et al., 2019; Tian et al., 2020), but they should train the larger network first, which leads to high training costs.”
LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPS (2023, background): “Many variants [3, 29, 40, 49, 57, 59, 68] have been proposed to achieve better knowledge transfer from the teacher to the student models.”
Novel Visual Category Discovery with Dual Ranking Statistics and Mutual Knowledge Distillation (2021, positive): “Knowledge distillation is often employed to learn a compact student model using the knowledge distilled from a larger teacher model by enforcing the agreement of outputs or representations between the two models ( e.g. , [24, 41, 54, 52, 27, 29, 1, 48]).”
Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation (2021, background): “Recent KD has evolved to give more importance to the KL divergence loss to improve performance when balancing the objective between CE loss and KL divergence loss [Hinton et al. , 2015; Tian et al. , 2019].”
Multi-label Contrastive Predictive Coding (2020, methodology): “We replace this objective with CPC and ML-CPC, using different choices of α that are fixed throughout training, and keeping the remaining hyperparameters identical to the CRD ones in [47].”
A Generalized Supervised Contrastive Learning Framework (2022, background): “In contrastive representation distillation [39], maximizations of a lower-bound to the mutual information between the teacher and student representations outperform distilling knowledge about class probabilities.”
Contrastive Representation Learning: A Framework and Review (2020, methodology): “Contrastive Representation Distillation (CDR) [100] uses a large, pre-trained teacher network as the encoder for both the positive and negative keys, while the queries are encoded by a small network learned to match the”