KDGAN: Knowledge Distillation with Generative Adversarial Networks

By Xiaojie Wang, Rui Zhang, Yu Sun, Jianzhong Qi. 2018

191 citations recorded by the source corpus.

Subfields: Generative Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

In Efficiency & Compression

Citations in the corpus

Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer (2019, methodology): “This is the main objective of knowledge transfer [21], [9], [43], [44], [54], [7], which is a powerful tool to share knowledge of a model through its predictions.”
Similarity-Preserving Knowledge Distillation (2019, background): “[34] introduced an additional discriminator network, and trained the student, teacher, and discrimina-”
Learning latent representations across multiple data domains using Lifelong VAEGAN (2020, methodology): “re non-dierentiable. In order to mitigate this, we use the Gumbel-Max trick [23,41] for achieving the dierentiable relaxation of discrete random variables. The Gumbel-softmax trick was also used in [17,27,40,63] and its capability of reducing the variation of gradients was shown in [63]. Accepted as”
Lifelong GAN: Continual Learning for Conditional Image Generation (2019, background): “Relevant applications include network compression [26] and network training acceleration [33].”
Improved Knowledge Distillation via Teacher Assistant (2019, methodology): “Adversarial methods also have been utilized for modeling knowledge transfer between teacher and student (Heo et al. 2018; Xu, Hsu, and Huang 2018; Wang et al. 2018b; 2018c).”
Self-Distillation: Towards Efficient and Compact Neural Networks (2021, methodology): “Wang et al. and Liu et al. apply knowledge distillation on objection detection, segmentation, depth prediction and other vision tasks [39], [49].”
Compression of Deep Learning Models for Text: A Survey (2020, methodology): “oss function deﬁned over classiﬁcation scores, and it contains information about both the distance and the path direction from the base sample to the decision boundary. Xu et al. [85] and Wang et al. [86] propose ideas based on Generative Adversarial Networks (GANs) for KD. In [85], the authors prop”
Teacher-Student Compression with Generative Adversarial Networks (2018, methodology): “Prime examples are the recent approaches of [39, 41, 38].”
Knowledge Representing: Efficient, Sparse Representation of Prior Knowledge for Knowledge Distillation (2019, methodology): “Furthermore, considering the sample bias is unavoidable, the generative adversarial networks for knowledge distillation (KDGAN) [30] is further used to produce a more robust posterior distribution for student classifier.”
Task-Oriented Feature Distillation (2020, methodology): “Besides image classiﬁcation, knowledge distillation has also been applied in other domains, such as object detection [63], image segmentation [37], nature language processing [8, 59, 28], distributed training [2], semi-supervised learning [32].”
Knowledge Distillation for Semantic Segmentation Using Channel and Spatial Correlations and Adaptive Cross Entropy (2020, background): “[25] proposed KDGAN, which applied generative adversarial networks (GAN) in the knowledge distillation method.”
Generative Adversarial Networks (GANs) (2020, methodology): “Generativediscriminative network pair (S12) Training of Single G (S12(i)) DCGAN [6], ProgressGAN [8], PacGAN [37], BayesianGAN [57], CapsNets [58], QuGANs [59], SAGAN [60] Training of Multiple Gs (S12(ii)) cGAN [44], AdaGAN [45], MAD-GAN [61], MGAN [62], MPM GAN [63], FictitiousGAN [64], MIX+GAN [65”
Improved Knowledge Distillation via Teacher Assistant: Bridging the Gap Between Student and Teacher (2019, methodology): “Another line of research focuses on distilling the original (large) network to a smaller network (Polino et al., 2018; Wang et al., 2018a), in which case the smaller network is more flexible in its architecture design does not have to be a sub-graph of the original network.”
Semi-Supervised Dialogue Policy Learning via Stochastic Reward Estimation (2020, methodology): “Semi-supervised learning aims to utilize unlabeled data to boost model performance, and is studied in computer vision (Iscen et al., 2019), item ranking (Park and Chang, 2019; Huang et al., 2019b), and multi-label classiﬁcation (Miyato et al., 2015; Wang et al., 2018, 2019b).”