Knowledge Distillation with Adversarial Samples Supporting Decision Boundary

By Byeongho Heo, Minsik Lee, Sangdoo Yun, J. Choi. 2018

160 citations recorded by the source corpus.

Many recent works on knowledge distillation have provided ways to transfer the knowledge of a trained network for improving the learning process of a new one, but finding a good technique for knowledge distillation is still an open problem. In this paper, we provide a new perspective based on a decision boundary, which is one of the most important component of a classifier. The generalization performance of a classifier is closely related to the adequacy of its decision boundary, so a good classifier bears a good decision boundary. Therefore, transferring information closely related to the decision boundary can be a good attempt for knowledge distillation. To realize this goal, we utilize an adversarial attack to discover samples supporting a decision boundary. Based on this idea, to transfer more accurate information about the decision boundary, the proposed algorithm trains a student classifier based on the adversarial samples supporting the decision boundary. Experiments show that the proposed method indeed improves knowledge distillation and achieves the state-of-the-arts performance.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Knowledge Distillation: A Survey (2020, methodology): “; Romero et al., 2015) and the loss functions for feature/distribution matching (Huang and Wang, 2017; Zagoruyko and Komodakis, 2016; Passalis and Tefas, 2018; Mirzadeh et al., 2019; Li et al., 2018; Heo et al., 2019b; Asif et al., 2019). The main advantage of oﬄine methods is simple and easy to be ”
Improved Knowledge Distillation via Teacher Assistant (2019, methodology): “For these the numbers are reported from the paper (Heo et al. 2018).”
Understanding and Improving Knowledge Distillation (2020, background): “…layer activations (Romero et al., 2014), auxiliary information (Vapnik & Izmailov, 2015), Jacobian matrix of the model parameters (Czarnecki et al., 2017; Srinivas & Fleuret, 2018), Gram matrix derived from pairs of layers (Yim et al., 2017), activation boundary (Heo et al., 2019), etc.”
Retrospective Adversarial Replay for Continual Learning (2022, background): “Training on such samples can help the model to learn about the boundaries between previous and current tasks and also retain the knowledge of previous tasks [7, 11, 23].”
Self-Distillation: Towards Efficient and Compact Neural Networks (2021, methodology): “Then, fruitful distillation methods have been proposed to transfer the knowledge of teacher models to student models based on feature maps [35], attention [36], the ﬂow of solution procedure [37], graph [38] and GAN-based methods [39], [40], [41].”
An Adversarial Feature Distillation Method for Audio Classification (2019, background): “[27] uses the adversarial learning strategy to induce student network training by using generated samples to effectively improve the ability of students to identify decision boundaries.”
Search for Better Students to Learn Distilled Knowledge (2020, background): “com [30], activation-based and gradient-based spatial attention maps [36], derivatives of the loss [5] and classification boundary [12].”
Collaborative Distillation in the Parameter and Spectrum Domains for Video Action Recognition (2020, background): “Knowledge distillation has been studied widely and shown feasibility in the image-level task (Thoker and Gall 2019; Hinton, Vinyals, and Dean 2015; Heo et al. 2019; Park et al. 2019), which aims to train small and efﬁcient video networks (student) with comparable performance as large ones (teacher).”
Extracurricular Learning: Knowledge Transfer Beyond Empirical Distribution (2020, result): “Our intuition is that uncertain samples are located close to the decision boundaries of the teacher model, similar to the argument in [27].”
Knowledge Representing: Efficient, Sparse Representation of Prior Knowledge for Knowledge Distillation (2019, methodology): “Through employing a penalized version [10, 13, 33] of final features of the teacher network, a joint learning is processed with the knowledge from posterior distribution.”
Compression of Deep Learning Models for Text: A Survey (2020, methodology): “to the adequacy of its decision boundary, so a good classiﬁer bears a good decision boundary. Therefore, transferring information closely related to the decision boundary can be a good attempt for KD [84]. To realize this goal, an adversarial attack is utilized to discover samples supporting a decis”
Exclusivity-Consistency Regularized Knowledge Distillation for Face Recognition (2020, background): “Heo et al. utilized an adversarial attack to discover supporting samples [14] and focused on the transfer of activation boundaries formed by hidden neurons [15] for knowledge distillation.”
Knowledge Distillation Beyond Model Compression (2020, methodology): “BSS [11] proposed a method for more explicitly matching the decision boundary by utilizing an adversarial attack to discover samples supporting a decision boundary and use an additional boundary supporting loss which encourages the student to match the output of the teacher on samples close”
Towards Oracle Knowledge Distillation with Neural Architecture Search (2019, background): “The main idea of KD is to transfer information from one model to another, and it is realized by learning a student network to mimic the output distributions of a teacher network (Hinton, Vinyals, and Dean 2015).”
Online Knowledge Distillation via Collaborative Learning (2020, background): “[9] transfers more accurate information via the route to the decision boundary.”