Deep Model Compression: Distilling Knowledge from Noisy Teachers

By Bharat Bhusan Sau, Vineeth Nallure Balasubramanian. 2016

192 citations recorded by the source corpus.

The remarkable successes of deep learning models across various applications have resulted in the design of deeper networks that can solve complex problems. How- ever, the increasing depth of such models also results in a higher storage and runtime complexity, which restricts the deployability of such very deep models on mobile and portable devices, which have limited storage and battery capacity. While many methods have been proposed for deep model compression in recent years, almost all of them have focused on reducing storage complexity. In this work, we extend the teacher-student framework for deep model com- pression, since it has the potential to address runtime and train time complexity too. We propose a simple method- ology to include a noise-based regularizer while training the student from the teacher, which provides a healthy im- provement in the performance of the student network. Our experiments on the CIFAR-10, SVHN and MNIST datasets show promising improvement, with the best performance on the CIFAR-10 dataset. We also conduct a comprehensive empirical evaluation of the proposed method under related settings on the CIFAR-10 dataset to show the promise of the proposed approach.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Wireless Network Intelligence at the Edge (2018, background): “Since noisy logit exchanges improve the KD performance when the noise level is properly adjusted [163], we expect exchanging Jacobian matrices to improve FD.”
Knowledge Distillation: A Survey (2020, neutral): “…knowledge transfer and explore the power of multiple teachers, several alternative methods have been proposed to simulate multiple teachers by adding different types of noise to a given teacher (Sau and Balasubramanian 2016) or by using stochastic blocks and skip connections (Lee et al. 2019c).”
Relational Knowledge Distillation (2019, background): “[29] propose a noise-based regularizer for KD while Lopes et al.”
Improved Knowledge Distillation via Teacher Assistant (2019, background): “To improve the student performance, Sau and Balasubramanian (2016) injected noise into teacher logits to make the student more robust.”
Sobolev Training for Neural Networks (2017, methodology): “This technique has many applications, such as network compression [21], ensemble merging [9], or more recently policy distillation in reinforcement learning [20].”
Agree to Disagree: Adaptive Ensemble Knowledge Distillation in Gradient Space (2020, methodology): “[ 24] used perturbed logits to simulate learning from multiple teachers while [6] argued to randomly select a teacher to strengthen the complementariness in the ensemble.”
Joint Training of Low-Precision Neural Network with Quantization Interval Parameters (2018, methodology): “Instead of temperature-based target generation[11; 13], we used the logit as target itself and minimize the mean-square error between the logits of the teacher and the student network as was done in [12].”
Stochasticity and Skip Connection Improve Knowledge Transfer (2021, background): “In other respect, skip connections let a residual network to be viewed as an ensemble of multiple paths of different lengths (Veit, Wilber, and Belongie 2016).”
Training Shallow and Thin Networks for Acceleration via Knowledge Distillation with Conditional Adversarial Networks (2017, methodology): “Variants of knowledge distillation has also been applied to many different tasks, such as semantic segmentation [31], pedestrian detection [35], face recognition [24], metric learning[6], reinforcement learning [38] and for regularization[34].”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, background): “To tackle these problems, some recent works [64], [182], [191], [219], [231], [234] propose data distillation by unifying data sources from multiple teachers as illustrated in Fig.”
Patient Knowledge Distillation for BERT Model Compression (2019, methodology): “Different from previous knowledge distillation methods (Hinton et al., 2015; Sau and Balasubramanian, 2016; Lu et al., 2017), we adopt a patient learning mechanism: instead of learning parameters from only the last layer of the teacher, we encourage the student model to extract knowledge also from p”
Computing Systems for Autonomous Driving: State of the Art and Challenges (2020, unknown): “In this case, collaborative training [203], model compression technologies [204], [205], [206], [207], [208], [209], and lightweight machine learning”
Adaptive multi-teacher multi-level knowledge distillation (2020, methodology): “[21] presents a noise-based regularization method, simulating learning from multiple teachers.”
Correlation Congruence for Knowledge Distillation (2019, background): “To improve the robustness of the student, Sau and Balasubramanian [27] perturbe the logits of a teacher as a regularization.”
Model Distillation with Knowledge Transfer from Face Classification to Alignment and Verification (2017, methodology): “Scratch Hidden Logits Soft [16] [21] [9] 60.”
Knowledge Transfer with Jacobian Matching (2018, background): “Sau & Balasubramanian (2016) found that adding noise to logits helps during teacher-student training.”
KDGAN: Knowledge Distillation with Generative Adversarial Networks (2018, methodology): “Consistent with the findings in existing work [23, 39], we also observe that ∇cLDS usually has a lower variance than ∇cLAD (see Figure 7 in Appendix D for empirical evidence that the variance of ∇cLDS is smaller than that of ∇cLAD during the training process).”