Feature-Level Ensemble Knowledge Distillation for Aggregating Knowledge from Multiple Networks

By Seonguk Park, Nojun Kwak. 2020

47 citations recorded by the source corpus.

. Knowledge Distillation (KD) aims to transfer knowledge in a teacher-student framework, by providing the predictions of the teacher network to the student network in the training stage to help the student network generalize better. It can use either a teacher with high capacity or an ensemble of multiple teachers. However, the latter is not convenient when one wants to use feature-map-based distillation methods. In this paper, we empirically show that using several non-linear transformation layer cope well with multiple-teacher setting compared to other kinds of feature-map-level distillation meth-ods. Comprehensively, this paper proposes a versatile and powerful training algorithm named FEature-level Ensemble knowledge Distil-lation (FEED), which aims to transfer the ensemble knowledge us-ing multiple teacher networks. In this study, we introduce a couple of training algorithms that transfer ensemble knowledge to the student at the feature-map-level. Among the feature-map-level distillation methods, using several non-linear transformations in parallel for transferring the knowledge of the multiple teachers helps the student ﬁnd more generalized solutions. We name this method as parallel FEED, and experimental results on CIFAR-100 and ImageNet show that our method has clear performance enhancements, without introducing any additional parameters or computations at test time. We also show the experimental results of sequentially feeding teacher’s information to the student, hence the name sequential FEED, and discuss the lessons obtained. Additionally, the empirical results on measuring the reconstruction errors at the feature map give hints for the enhancements.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Knowledge Distillation: A Survey (2020, background): “Several multi-teacher knowledge distillation methods have recently been proposed (Sau and Balasubramanian 2016; You et al. 2017; Chen et al. 2019b; Furlanello et al. 2018; Yang et al. 2019a; Zhang et al. 2018b; Lee et al. 2019c; Park and Kwak 2020; Papernot et al. 2017; Fukuda et al. 2017; Ruder et ”