Knowledge distillation using unlabeled mismatched images

By Mandar Kulkarni, Kalpesh Patil, Shirish Subhash Karande. 2017

17 citations recorded by the source corpus.

Current approaches for Knowledge Distillation (KD) either directly use training data or sample from the training data distribution. In this paper, we demonstrate effectiveness of 'mismatched' unlabeled stimulus to perform KD for image classification networks. For illustration, we consider scenarios where this is a complete absence of training data, or mismatched stimulus has to be used for augmenting a small amount of training data. We demonstrate that stimulus complexity is a key factor for distillation's good performance. Our examples include use of various datasets for stimulating MNIST and CIFAR teachers.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Scalable Recollections for Continual Lifelong Learning (2017, methodology): “Additionally, unlabelled data has been widely used (Riemer et al., 2016; Laine & Aila, 2017; Ao et al., 2017; Kulkarni et al., 2017) for knowledge distillation.”
Few-shot learning of neural networks from scratch by pseudo example optimization (2018, background): “Other previous researches demonstrates that knowledge can be transferred from either intermediate layers of a teacher model [28, 37], from multiple teacher models [38], mismatched unsupervised stimuli [20] or examples reconstructed from meta-data [10].”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, background): “[112] create a ‘mismatched’ unlabeled stimulus (e.”
Dream Distillation: A Data-Independent Model Compression Framework (2019, methodology): “shown to work with unlabeled datasets (Kulkarni et al., 2017).”
Imitation networks: Few-shot learning of neural networks from scratch (2018, background): “…knowledge can be transferred from either intermediate layers of a teacher model [Romero et al., 2015; Yim et al., 2017], from multiple teacher models [You et al., 2017], mismatched unsupervised stimuli [Kulkarni et al., 2017] or examples reconstructed from meta-data [Gontijo Lopes et al., 2017].”
Knowledge Distillation for Optimization of Quantized Deep Neural Networks (2019, background): “Leveraging the knowledge contained in previously trained networks has attracted attention in many applications for model compression [8, 9, 10, 11] and learning algorithms [12, 13, 14, 15].”
Generative Knowledge Distillation for General Purpose Function Compression (2017, background): “Additionally, unlabelled data has been widely used [26, 16, 1, 15] for knowledge distillation.”
Empirical Analysis of Knowledge Distillation Technique for Optimization of Quantized Deep Neural Networks (2019, methodology): “…improvement of the knowledge distillation technique, it utilizes in variety of applications (Chebotar and Waters 2016; Dai and Van Gool 2018; Chen et al. 2017; Oord et al. 2017) and learning algorithms (Romero et al. 2014; Kulkarni, Patil, and Karande 2017; Park et al. 2019; Yim et al. 2017).”