KTAN: Knowledge Transfer Adversarial Network
By Peiye Liu, Wu Liu, Huadong Ma, Tao Mei, Mingoo Seok. 2018
29 citations recorded by the source corpus.
Knowledge distillation was pioneered to transfer the generalization ability of a large teacher deep network to a light-weight student network. The student network can retain the high quality of the teacher network, yet exhibiting low computational complexity and storage requirement, which is attractive for deploying a deep convolution neural network on a resource-constrained mobile device. However, most of the existing methods focus on transferring the probability distribution of a softmax layer in a teacher network and neglect the intermediate representations. However, we find that the intermediate representation is critical for a student network to better understand the transferred generalization as compared to the probability distribution only. In this paper, therefore, we propose such a knowledge transfer adversarial network method which holistically considers both intermediate representations and probability distributions of a teacher network. To transfer the knowledge of intermediate representations, we set high-level teacher feature maps as a target, toward which the method trains student feature maps. Furthermore, to support various structures of a student network, we arrange a novel teacher-to-student layer. Finally, the proposed method employs an adversarial learning process. Specifically, it includes a discriminator network to fully exploit the spatial correlation of feature maps during the training process of a student network. The experimental results demonstrate that the proposed method can significantly improve the performance of a student network on two important vision tasks, image classification and object detection.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
Citations in the corpus
- Knowledge Distillation: A Survey (2020, methodology): “…are other special interesting applications of knowledge distillation, such as neural architecture search (Macko et al. 2019; Bashivanetal.2019),interpretabilityofdeepneuralnetworks (Liu et al. 2018b), and federated learning (Bistritz et al. 2020; Lin et al. 2020; Seo et al. 2020; He et al. 2020a).”
- Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation (2019, background): “sue from the perspective of attention mechanism, attempting to align the features of attention regions. Furthermore, some researchers extended knowledge distillation to generative adversarial problem [33, 27]. In the other domains, knowledge distillation also shows its potential. Furlanello et al. i”
- Knowledge Squeezed Adversarial Network Compression (2019, unknown): “Various derivatives have been implemented in different tasks: object detection [31], semantic segmentation [32], and metric learning [33], etc.”
- Self-Distillation: Towards Efficient and Compact Neural Networks (2021, methodology): “Then, fruitful distillation methods have been proposed to transfer the knowledge of teacher models to student models based on feature maps [35], attention [36], the flow of solution procedure [37], graph [38] and GAN-based methods [39], [40], [41].”
- Feature-map-level Online Adversarial Knowledge Distillation (2020, methodology): “While KD only utilizes the logit, method such as FitNet (Romero et al., 2014), AT (Zagoruyko & Komodakis, 2016a), FT (Kim et al., 2018) and KTAN (Liu et al., 2018) use the intermediate feature representation to transfer the knowledge of a teacher network.”
- Distilling Object Detectors with Task Adaptive Regularization (2020, methodology): “Similar works are presented in [31,19,6] but are designed mainly for classifiers.”