Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons

By Byeongho Heo, Minsik Lee, Sangdoo Yun, J. Choi. 2018

623 citations recorded by the source corpus.

An activation boundary for a neuron refers to a separating hyperplane that determines whether the neuron is activated or deactivated. It has been long considered in neural networks that the activations of neurons, rather than their exact output values, play the most important role in forming classificationfriendly partitions of the hidden feature space. However, as far as we know, this aspect of neural networks has not been considered in the literature of knowledge transfer. In this paper, we propose a knowledge transfer method via distillation of activation boundaries formed by hidden neurons. For the distillation, we propose an activation transfer loss that has the minimum value when the boundaries generated by the student coincide with those by the teacher. Since the activation transfer loss is not differentiable, we design a piecewise differentiable loss approximating the activation transfer loss. By the proposed method, the student learns a separating boundary between activation region and deactivation region formed by each neuron in the teacher. Through the experiments in various aspects of knowledge transfer, it is verified that the proposed method outperforms the current state-of-the-art.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Few-Shot Class-Incremental Learning via Relation Knowledge Distillation (2021, positive): “(2) Feature-based knowledge approaches: Feature layer as intermediate representations is used to train the student network (Romero et al. 2014; Zagoruyko and Komodakis 2016; Heo et al. 2019).”
Knowledge Distillation: A Survey (2020, methodology): “; Romero et al., 2015) and the loss functions for feature/distribution matching (Huang and Wang, 2017; Zagoruyko and Komodakis, 2016; Passalis and Tefas, 2018; Mirzadeh et al., 2019; Li et al., 2018; Heo et al., 2019b; Asif et al., 2019). The main advantage of oﬄine methods is simple and easy to be ”
Distilling portable Generative Adversarial Networks for Image Translation (2020, neutral): “…an alternative approach to learn a portable student network with comparable performance and fewer parameters by inheriting knowledge from the teacher network (Hinton, Vinyals, and Dean 2015; Romero et al. 2014; You et al. 2017; Wang et al. 2018b; Heo et al. 2019), i.e. pre-trained heavy network.”
Differentiable Feature Aggregation Search for Knowledge Distillation (2020, methodology): “We compare our method with eight single-teacher distillation methods: KD [13], FitNets [30], AT [38], Jacobian [31], FT [16], AB [12], SP [33] and Margin [11].”
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review (2023, methodology): “The student model can also use other information from the teacher model, such as activations [183], neurons [184], or features of intermediate layers [185] as knowledge to improve the performance.”
Distilling Object Detectors via Decoupled Features (2021, background): “The information from soft label outputs [24, 28, 38, 13] or intermediate features [1, 23, 66] of a well-optimized teacher network have been well exploited to learn the student networks, but these methods cannot be directly extended to the detection task which needs to further figure out where the ob”
General Instance Distillation for Object Detection (2021, neutral): “Those knowledge for distillation can be classified into the following categories [7]: response-based [10], feature-based [27, 12, 9, 1, 35], and relation-based [22, 20, 31, 33].”
LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPS (2023, background): “Many variants [3, 29, 40, 49, 57, 59, 68] have been proposed to achieve better knowledge transfer from the teacher to the student models.”
Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labels (2021, background): “Studies on KD have enriched and diversified the options for the teacher, such as feature map distillation [58, 19, 18], relation-based distillation [34, 47], ensemble distillation [40, 61], or iterative self-distillation [9, 55, 53].”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “[63] use the original teacher’s feature in the form of binarized values, namely via a separating hyperplane (activation boundary (AB)) that determines whether neurons are activated or deactivated.”
Balanced Knowledge Distillation for Long-tailed Learning (2021, background): “Apart from the final prediction [35], other types of knowledge, like the intermediate representations [41, 24, 15] or the relationships between different layers [52] or data samples [54, 37, 32], can also be used to guide the learning of the student model.”