Paraphrasing Complex Network: Network Compression via Factor Transfer

By Jangho Kim, Seonguk Park, Nojun Kwak. 2018

652 citations recorded by the source corpus.

Many researchers have sought ways of model compression to reduce the size of a deep neural network (DNN) with minimal performance degradation in order to use DNNs in embedded systems. Among the model compression methods, a method called knowledge transfer is to train a student network with a stronger teacher network. In this paper, we propose a novel knowledge transfer method which uses convolutional operations to paraphrase teacher's knowledge and to translate it for the student. This is done by two convolutional modules, which are called a paraphraser and a translator. The paraphraser is trained in an unsupervised manner to extract the teacher factors which are defined as paraphrased information of the teacher network. The translator located at the student network extracts the student factors and helps to translate the teacher factors by mimicking them. We observed that our student network trained with the proposed factor transfer method outperforms the ones trained with conventional knowledge transfer methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Contrastive Representation Distillation (2019, background): “Other papers (Yim et al., 2017; Huang & Wang, 2017; Kim et al., 2018; Yim et al., 2017; Huang & Wang, 2017; Ahn et al., 2019; Koratana et al., 2019) have enforced various criteria based on representations.”
FEED: Feature-level Ensemble for Knowledge Distillation (2019, methodology): “Additionally, Kim (Kim, Park, and Kwak 2018) state that they resolve the ‘inherent difference’ between two networks.”
Class-Incremental Learning with Cross-Space Clustering and Controlled Transfer (2022, background): “Several other papers have enforced criteria based on multi-layer representations of a network [53, 20, 24, 3, 28].”
Knowledge Distillation: A Survey (2020, methodology): “017) using neuron selectivity transfer. Passalis and Tefas (2018) transferred knowledge by matching the probability distribution in feature space. To make it easier to transfer the teacher knowledge, Kim et al. (2018) introduced so called “factors” as a more understandable form of intermediate repre”
Localization Distillation for Object Detection (2022, background): “It has been widely validated to be useful for boosting the performance of small-sized student networks by transferring the generalized knowledge captured by large-sized teacher networks [24], [29], [31], [58], [63], [78].”
Multi-label Contrastive Predictive Coding (2020, background): “Two baselines are considered: the original KD objective in [20] and the state-of-the-art CRD objective in [47], since other baselines [29, 1, 23, 26] are shown to have inferior performance in general.”
Distilling Knowledge via Knowledge Review (2021, methodology): “Following previous notations, single-layer knowledge distillation can be represented as where M is transformation that transfers the feature to target representation of attention maps [38] or factors [14].”
An Adversarial Feature Distillation Method for Audio Classification (2019, methodology): “proposed using feature-based distillation [25] instead of soft label based distillation, the structure shown in Figure 1(b).”
Novel Visual Category Discovery with Dual Ranking Statistics and Mutual Knowledge Distillation (2021, neutral): “Knowledge distillation is often employed to learn a compact student model using the knowledge distilled from a larger teacher model by enforcing the agreement of outputs or representations between the two models ( e.g. , [24, 41, 54, 52, 27, 29, 1, 48]).”
Ensemble Knowledge Distillation for Learning Improved and Efficient Networks (2019, methodology): “Other methods explored knowledge distillation using activation maps [9], attention maps [25], Jacobians [20], and unsupervised feature factors [14].”
Learning an Evolutionary Embedding via Massive Knowledge Distillation (2020, methodology): “Besides, in Table 2, EEL also outperforms other state-of-the-art methods, including KD [16], Logits [1], Fitnet [50], Attention Transfer (AT) [69], Flow of Solution Procedure (FSP) [67], Neural Selective Transfer (NST) [21], Probabilistic Knowledge Transfer (PKT) [46], Factor Transfer (FT) [24], Dee”
Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation (2021, background): “One attempted to distill not only the softened probabilities of the teacher network but also the hidden feature vector so that the student could be trained with rich information from the teacher [Romero et al., 2014; Zagoruyko and Komodakis, 2016a; Srinivas and Fleuret, 2018; Kim et al., 2018; Heo e”
Lipschitz Continuity Guided Knowledge Distillation (2021, background): “Setup Teacher Baseline KD FitNets AT Jacobian FT AB OFD AFD LONDON [16] [34] [48] [35] [21] [15] [14] [41] (ours) (a) 21.”
Feature-map-level Online Adversarial Knowledge Distillation (2020, methodology): “While KD only utilizes the logit, method such as FitNet (Romero et al., 2014), AT (Zagoruyko & Komodakis, 2016a), FT (Kim et al., 2018) and KTAN (Liu et al., 2018) use the intermediate feature representation to transfer the knowledge of a teacher network.”
Subclass Distillation (2020, methodology): “Several distillation methods have been proposed in the last few years (Zagoruyko & Komodakis, 2016; Tung & Mori, 2019; Peng et al., 2019; Ahn et al., 2019; Park et al., 2019; Passalis & Tefas, 2018; Heo et al., 2018; Kim et al., 2018; Yim et al., 2017; Huang & Wang, 2017).”
Show, Attend and Distill: Knowledge Distillation via Attention-based Feature Matching (2021, neutral): “Kim et al. (Kim, Park, and Kwak 2018) extracted low-dimensional representations of the features via multiple auto-encoders and transfer them to the student.”
Learning Compatible Embeddings (2021, background): “Various knowledge sources are adopted in literature such as classification logits [17, 1, 28], hint layers [35, 54, 49, 3] and multi-layer groups [52, 51, 20].”