Training Shallow and Thin Networks for Acceleration via Knowledge Distillation with Conditional Adversarial Networks

By Zheng Xu, Yen-Chang Hsu, Jiawei Huang. 2017

114 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Knowledge Distillation: A Survey (2020, methodology): “Inspired by this, more and more adversarial knowledge distillation methods have been proposed to enable the teacher/student networks to have a better understanding of the true data distribution (Wang et al., 2018d; Xu et al., 2017; Micaelli and Storkey, 2019; Xu et al., 2018; Liu et al., 2018b; Wang”
Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion (2019, background): “Knowledge distillation and its improved variants [1, 53, 57, 67, 73] enable teaching students with goals such as quantization [42, 55], compact neural network architecture design [57], semantic segmentation [31], self-distillation [14], and un-/semi-supervised learning [34, 54, 70].”
Improved Knowledge Distillation via Teacher Assistant (2019, methodology): “Adversarial methods also have been utilized for modeling knowledge transfer between teacher and student (Heo et al. 2018; Xu, Hsu, and Huang 2018; Wang et al. 2018b; 2018c).”
Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher (2020, background): “[23] and [25] adopt adversarial training to reduce the difference of label distribution between teacher and student.”
Complementary Relation Contrastive Distillation (2021, background): “Knowledge distillation aims to transfer the knowledge from one deep learning model (the teacher) to another (the student), such as distilling a large network into a smaller one [19, 49, 2, 48, 12] or ensembling a collection of models into a single model [29, 37, 27, 45].”
Compression of Deep Learning Models for Text: A Survey (2020, unknown): “In [85], the authors propose a method under which the knowledge is transferred from teacher to student through a discriminator in the GAN-based approach.”
Intra-class Feature Variation Distillation for Semantic Segmentation (2020, background): “Adversarial learning for knowledge distillation can be ﬁrst found in [38].”
Knowledge Distillation with Feature Maps for Image Classification (2018, methodology): “In [34], Xu & Huang proposed the method that uses conditional adversarial networks to make student networks learn the logits of the teacher networks.”
Task-Oriented Feature Distillation (2020, methodology): “Later, abundant feature distillation methods are proposed to encourage the student models to mimic the features of teacher models [53, 62, 66, 7, 20].”
Correlation Congruence for Knowledge Distillation (2019, methodology): “In [35], they adopt generative adversarial networks combined with distillation to learn the loss function to better transfer teacher’s knowledge to student.”
GAN-Knowledge Distillation for One-Stage Object Detection (2019, methodology): “Ensemble via Adversarial Learning), [33] proposed by Xu et al.”
Generative Adversarial Networks (GANs) (2020, methodology): “KDGAN idea is inspired by [110] in which D is used to train C for learning the data distribution produced by T, while in KDGAN, D trains C to learn the real data distribution directly.”