A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning

By Junho Yim, Donggyu Joo, Ji‐Hoon Bae, Junmo Kim. 2017

1,725 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need? (2020, background): “In was shown in [63] that KD has several benefits for optimization and knowledge transfer between tasks.”
Born Again Neural Networks (2018, methodology): “Key differences Our work differs from (Yim et al., 2017) in several key ways.”
Contrastive Representation Distillation (2019, background): “Other papers (Yim et al., 2017; Huang & Wang, 2017; Kim et al., 2018; Yim et al., 2017; Huang & Wang, 2017; Ahn et al., 2019; Koratana et al., 2019) have enforced various criteria based on representations.”
FEED: Feature-level Ensemble for Knowledge Distillation (2019, background): “Yim (Yim et al. 2017) introduced another knowledge transfer technique for faster optimization and applied it also to transfer learning.”
Similarity-Preserving Knowledge Distillation (2019, methodology): “Several prior alternatives [12, 31, 38, 41] are described in the introduction and some key differences are highlighted in Section 2.”
Variational Information Distillation for Knowledge Transfer (2019, methodology): “Recent works [3, 6, 13, 30, 31] relax the regularization of matching the entire layer by matching carefully designed features/statistics extracted from intermediate layers of the teacher and the student networks, e.g ., attention maps [31] and maximum mean discrepancy [13].”
Paraphrasing Complex Network: Network Compression via Factor Transfer (2018, methodology): “[28] defined the flow of solution procedure (FSP) matrix calculated by Gram matrix of feature maps from two layers in order to transfer knowledge.”
Network Pruning via Transformable Architecture Search (2019, background): “The parameters of the searched/pruned networks are then learned by knowledge transfer [18, 41, 43].”
Class-Incremental Learning with Cross-Space Clustering and Controlled Transfer (2022, background): “Several other papers have enforced criteria based on multi-layer representations of a network [53, 20, 24, 3, 28].”
Transfer without Forgetting (2022, background): “Alternatively, transfer can be induced via multi-level Knowledge Distillation, guided by meta-learning [30], attention [66] or higher-level descriptions of the flow of information within the model [67].”
Few-Shot Class-Incremental Learning via Relation Knowledge Distillation (2021, background): “(3) Relation-based knowledge further explores the structures between different layers and data samples (Yim et al. 2017; Park et al. 2019; Ramakrishnan et al. 2020).”
Knowledge Distillation: A Survey (2020, methodology): “prediction knowledge at classiﬁer level and representation graph distills the feature knowledge from pairwise ensemble representations of the compact bilinear pooling layers of teachers . FSP matrix (Yim et al., 2017; Lee et al., 2018) The relations of any two feature maps from any two layers of net”
Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion (2019, methodology): “These methods and improved variants [1, 53, 57, 67, 73] enable teaching students with goals such as quantization [42, 55], compact neural network architecture design [57], semantic segmentation [31], self-distillation [14], and un-/semi-supervised learning [34, 54, 70].”
Multilingual Neural Machine Translation with Knowledge Distillation (2019, background): “…of multiple models), and then train a small/shallow student model to mimic the behaviors of the teacher model, such as its hidden representation (Yim et al., 2017; Romero et al., 2014), its output probabilities (Hinton et al., 2015; Freitag et al., 2017) or directly training on the sentences…”
Well-Read Students Learn Better: The Impact of Student Initialization on Knowledge Distillation (2019, background): “Surprisingly, student model initialization has received relatively little attention while a substantial body of work has focused on specific student architectures and novel transfer methods (Ba & Caruana, 2014; Romero et al., 2014; Kim & Rush, 2016; Yim et al., 2017; Hu et al., 2018).”
Relational Knowledge Distillation (2019, methodology): “Likewise, many other methods [1, 2, 8, 12, 25, 45, 47] can also be formulated as a form of Eq.”
A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks (2020, neutral): “The use of knowledge distillation in an IL context is similar to self-distillation [12, 61] in that it operates with the same network architecture for the teacher and the student.”
Data-Free Learning of Student Networks (2019, positive): “[29] introduced the FSP (Flow of Solution Procedure) matrix to inherit the relationship between features from two layers.”
Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed (2021, background): “Knowledge distillation (Bucilua et al., 2006; Hinton et al., 2015) has been explored extensively in supervised learning tasks such as image classiﬁcation [26, 46, 44, 7, 36, 43], language modeling [35, 37, 28, 14, 34], and speech recognition [3, 6, 40, 17, 9].”