Feature-map-level Online Adversarial Knowledge Distillation

By Inseop Chung, Seonguk Park, Jangho Kim, Nojun Kwak. 2020

155 citations recorded by the source corpus.

Feature maps contain rich information about image intensity and spatial correlation. However, previous online knowledge distillation methods only utilize the class probabilities. Thus in this paper, we propose an online knowledge distillation method that transfers not only the knowledge of the class probabilities but also that of the feature map using the adversarial training framework. We train multiple networks simultaneously by employing discriminators to distinguish the feature map distributions of different networks. Each network has its corresponding discriminator which discriminates the feature map from its own as fake while classifying that of the other network as real. By training a network to fool the corresponding discriminator, it can learn the other network's feature map distribution. We show that our method performs better than the conventional direct alignment method such as L1 and is more suitable for online distillation. Also, we propose a novel cyclic learning scheme for training more than two networks together. We have applied our method to various network architectures on the classification task and discovered a significant improvement of performance especially in the case of training a pair of a small network and a large one.

Subfields: Artificial Intelligence, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Lin Wang
3. Kuk-Jin Yoon
4. Jianping Gou
5. S. Maybank
6. Baosheng Yu
7. Nojun Kwak
8. Chen Qian

In Radiology, Nuclear Medicine and Imaging

1. Lin Wang
2. Kuk-Jin Yoon
3. Dacheng Tao
4. Eric P. Xing
5. S. Maybank
6. Zhiqiang Shen
7. Fei Wang
8. Baosheng Yu

Citations in the corpus

• Knowledge Distillation: A Survey (2020, methodology): “y, an online adversarial knowledge distillation method is proposed to simultaneously train multiple networks by the discriminators using knowledge from both the class probabilities and a feature map (Chung et al., 2020). 4.3 Self-Distillation In self-distillation, the same networks are used for the ”
• Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillation (2021, background): “where α and β are hyperparameters; and we choose α ∈ [1, 2, 3] and β ∈ [100, 200]; further details are explained in Appendix Sensitivity Analysis.”
• Local Correlation Consistency for Knowledge Distillation (2020, methodology): “[2] try to capture the consistent feature map of intermediate layers by the adversarial learning.”
• A Fast Knowledge Distillation Framework for Visual Recognition (2021, background): “Recently, many variants and extensions are proposed [6, 18, 24–26, 35, 37, 44, 47, 49, 54], such as employing internal feature representations [30], adversarial training with discriminators [32], transfer flow [48], contrastive distillation [40], patient and consistent [2] etc.”
• Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “Method GAN type Purpose Inputs of D Number of D Online KD Chen [31] First type Classification Logits One No Belagiannis [19] First type Classification Features One No Liu [135] First type ClassificationObject detection Features One No Hong [89] First type Object detection Features Six No Wang [229] ”
• Feature Fusion for Online Mutual Knowledge Distillation (2019, background): “Besides probability distribution, some other researches have tried to distill various features to the student [9], [10], [11], [12], [13], [14], [15].”