Adversarial Learning of Portable Student Networks

By Yunhe Wang, Chang Xu, Chao Xu, Dacheng Tao. 2018

76 citations recorded by the source corpus.

Effective methods for learning deep neural networks with fewer parameters are urgently required, since storage and computations of heavy neural networks have largely prevented their widespread use on mobile devices. Compared with algorithms which directly remove weights or filters for obtaining considerable compression and speed-up ratios, training thin deep networks exploiting the student-teacher learning paradigm is more flexible. However, it is very hard to determine which formulation is optimal to measure the information inherited from teacher networks. To overcome this challenge, we utilize the generative adversarial network (GAN) to learn the student network. In practice, the generator is exactly the student network with extremely less parameters and the discriminator is used as a teaching assistant for distinguishing features extracted from student and teacher networks. By simultaneously optimizing the generator and the discriminator, the resulting student network can produce features of input data with the similar distribution as that of features of the teacher network. Extensive experimental results on benchmark datasets demonstrate that the proposed method is capable of learning well-performed portable networks, which is superior to the state-of-the-art methods.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

AutoGAN-Distiller: Searching to Compress Generative Adversarial Networks (2020, neutral): “These encoder-decoder GANs also gain popularity in image enhancement (Wang et al., 2018a; Kupyn et al., 2019; Jiang et al., 2019) and editing (Sanakoyeu et al., 2018; Yang et al., 2019).”
Knowledge Distillation: A Survey (2020, methodology): “…extended to other visual recognition applications, including face recognition (Luo et al. 2016; Kong et al. 2019; Yan et al. 2019; Ge et al. 2018; Wang et al. 2018b, 2019c; Duong et al. 2019; Wu et al. 2020; Wang et al. 2017; Zhang et al. 2020b; Wang et al. 2020b), image/video segmentation (He…”
Improved Knowledge Distillation via Teacher Assistant (2019, methodology): “Adversarial methods also have been utilized for modeling knowledge transfer between teacher and student (Heo et al. 2018; Xu, Hsu, and Huang 2018; Wang et al. 2018b; 2018c).”
Low-resolution Visual Recognition via Deep Feature Distillation (2019, background): “, directly learning a student network with the indication of a teacher network which consists of more valuable information [17, 18, 19, 20].”
Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inference (2020, methodology): “Layer factorization (Tai et al., 2016; Yu et al., 2017), quantization (Wu et al., 2016), model distillation (Wang et al., 2018c) and weight sharing (Wu et al., 2018) have also been respectively found effective.”
Learning Student Networks via Feature Embedding (2018, methodology): “[26] exploited generative adversarial networks to make feature distribution of teacher and student networks similar.”
Amalgamating Knowledge from Two Teachers for Task-oriented Dialogue System with Adversarial Training (2020, background): “A typical application is to transfer knowledge from a large, powerful teacher network to a compact yet accurate student network, so as to boost the training process and the resulting performance (Watanabe et al., 2017; Bucilu and Niculescu-Mizil, 2006; Wang et al., 2018).”
Towards Evolutionary Compression (2018, background): “[43] proposed to exploit adversarial loss to supervise the compression.”
MixSKD: Self-Knowledge Distillation from Mixup for Image Recognition (2022, methodology): “Motivated by [36], we propose an adversarial feature Self-KD method that utilizes l2 loss to force the feature maps to be close and a discriminator loss to simultaneously increase the difficulty of mimicry.”
Teacher-Student Compression with Generative Adversarial Networks (2018, methodology): “These differ from standard TSC by employing non-standard GAN-type compression losses, in which the student acts as the discriminator (Wang et al., 2018b) or generator (Xu et al., 2018; Wang et al., 2018a); Wang et al. (2018a) also train the teacher and student together.”
ShrinkTeaNet: Million-scale Lightweight Face Recognition via Shrinking Teacher-Student Networks (2019, methodology): “Factors `1 3 Act_Bound [11] Closed-set Binarization 1× 1 conv Act Map Marginal `2 3 AL_PSN[14] Closed-set Identity Identity Feat.”
Knowledge Squeezed Adversarial Network Compression (2019, background): “Motivations: 1) To inherit the information from teacher network, the aforementioned methods [14, 15, 13] usually focus on result-oriented learning, which utilizes the adversarial training to minimize the discrepancy between distributions of extracted feature (i.”
Distilling portable Generative Adversarial Networks for Image Translation (2020, unknown): “…an alternative approach to learn a portable student network with comparable performance and fewer parameters by inheriting knowledge from the teacher network (Hinton, Vinyals, and Dean 2015; Romero et al. 2014; You et al. 2017; Wang et al. 2018b; Heo et al. 2019), i.e. pre-trained heavy network.”