Variational Information Distillation for Knowledge Transfer

By Sungsoo Ahn, Shell Xu Hu, Andreas C. Damianou, Neil D. Lawrence, Zhenwen Dai. 2019

763 citations recorded by the source corpus.

Transferring knowledge from a teacher neural network pretrained on the same or a similar task to a student neural network can significantly improve the performance of the student neural network. Existing knowledge transfer approaches match the activations or the corresponding hand-crafted features of the teacher and the student networks. We propose an information-theoretic framework for knowledge transfer which formulates knowledge transfer as maximizing the mutual information between the teacher and the student networks. We compare our method with existing knowledge transfer methods on both knowledge distillation and transfer learning tasks and show that our method consistently outperforms existing methods. We further demonstrate the strength of our method on knowledge transfer across heterogeneous network architectures by transferring knowledge from a convolutional neural network (CNN) to a multi-layer perceptron (MLP) on CIFAR-10. The resulting MLP significantly outperforms the-state-of-the-art methods and it achieves similar performance to the CNN with a single convolutional layer.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Contrastive Representation Distillation (2019, methodology): “Other papers (Yim et al., 2017; Huang & Wang, 2017; Kim et al., 2018; Yim et al., 2017; Huang & Wang, 2017; Ahn et al., 2019) have enforced various criteria based on representations.”
Zero-shot Knowledge Transfer via Adversarial Belief Matching (2019, methodology): “58% if we use the student loss described in Equation 1, which is 2% better than VID’s performance when it uses an extra M = 100 images per class.”
Class-Incremental Learning with Cross-Space Clustering and Controlled Transfer (2022, background): “Several other papers have enforced criteria based on multi-layer representations of a network [53, 20, 24, 3, 28].”
Novel Class Discovery without Forgetting (2022, methodology): “Numerically computing exact mutual information is intractable, and hence we resort to a variational approximation q(l|u) to true distribution p(l|u) [2,3] as follows:”
Knowledge Distillation: A Survey (2020, background): “The activations, neurons or features of intermediate layers also can be used as the knowledge to guide the learning of the student model (Romero et al., 2015; Huang and Wang, 2017; Ahn et al., 2019; Heo et al., 2019c; Zagoruyko and Komodakis, 2017).”
Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion (2019, background): “Knowledge distillation and its improved variants [1, 53, 57, 67, 73] enable teaching students with goals such as quantization [42, 55], compact neural network architecture design [57], semantic segmentation [31], self-distillation [14], and un-/semi-supervised learning [34, 54, 70].”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “Model distillation [3, 94, 148, 173, 189, 200, 260, 262, 280] is a method in which a large model with higher accuracy is used as a teacher to help the training of a compact student model.”
Online Knowledge Distillation with Diverse Peers (2019, background): “Several recent works attempt to further improve the performance with new formulations of teacher-learned knowledge (Yim et al. 2017; Chen, Zhang, and Dong 2018; Ahn et al. 2019).”
Compressing Visual-linguistic Model via Knowledge Distillation (2021, background): “Typically, the knowledge is transferred though mimicking the output logit [25, 57, 17], reducing the divergence of feature maps [80, 27, 78], or learning the intermediate layer representations [36, 1], etc.”
Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANs (2020, background): “Investigating more advanced techniques ( e.g ., [1, 18, 34]) would be an interesting and promising future direction.”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “On the contrary, different from the transformation of teacher, FitNet [85], AB [83], Heo et al. [82] and VID [5] do to change the dimension of teacher’s feature representations and design TF s with a ‘bottleneck’ layer (1×1 convolution) to make student’s feature to match the dimension with the teach”
CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information (2020, background): “, 2016), information distillation (Ahn et al., 2019), and reinforcement learning (Florensa et al.”
Dual-Teacher Class-Incremental Learning With Data-Free Generative Replay (2021, methodology): “As proposed in [1], we take the variational lower bound of the mutual information and perform variational information maximization [3] with a Gaussian prior.”
Cross-Layer Distillation with Semantic Calibration (2020, methodology): “However, existing efforts mainly focus on particular representations of feature maps to capture the enriched knowledge and enable knowledge transfer based on handcrafted layer assignments, such as random selection or oneto-one association (Romero et al. 2015; Zagoruyko and Komodakis 2017; Ahn et al.”
Complementary Relation Contrastive Distillation (2021, methodology): “06 VID [1] 75.”
Training convolutional neural networks with cheap convolutions and online distillation (2019, background): “The extra supervision extracted from a pre-trained teacher model is often in form of class posterior probabilities [30], feature representations [31, 32, 40, 4], distribution of intermediate activations [41, 42], or inter-layer flow [43]).”