Knowledge Transfer with Jacobian Matching

By Suraj Srinivas, Franccois Fleuret. 2018

181 citations recorded by the source corpus.

Classical distillation methods transfer representations from a "teacher" neural network to a "student" network by matching their output activations. Recent methods also match the Jacobians, or the gradient of output activations with the input. However, this involves making some ad hoc decisions, in particular, the choice of the loss function. In this paper, we first establish an equivalence between Jacobian matching and distillation with input noise, from which we derive appropriate loss functions for Jacobian matching. We then rely on this analysis to apply Jacobian matching to transfer learning by establishing equivalence of a recent transfer learning procedure to distillation. We then show experimentally on standard image datasets that Jacobian-based penalties improve distillation, robustness to noisy inputs, and transfer learning.

Subfields: Artificial Intelligence, Computer Networks and Communications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

Citations in the corpus

Wireless Network Intelligence at the Edge (2018, background): “2) Federated Jacobian Distillation (FJD) Model inputoutput Jacobian matching is interpreted as a KD operation that inserts noise into logits [170].”
Knowledge Distillation: A Survey (2020, methodology): “…well reﬂect the neuron activations of convolutional neural network, some attention mechanisms are used in knowledge distillation to improve the performance of the student network (Zagoruyko and Komodakis 2017; Huang and Wang 2017; Srinivas and Fleuret 2018; Crowley et al. 2018; Song et al. 2018).”
Neural probabilistic motor primitives for humanoid control (2018, methodology): “Knowledge transfer refers to the broad class of approaches which transfer the input-output functional mapping, to some extent or another, from a teacher (or expert ) to a student (Hinton et al., 2015; Srinivas & Fleuret, 2018; Furlanello et al., 2018).”
Self-Distillation Amplifies Regularization in Hilbert Space (2020, background): “…proposed, where instead of output predictions, the student matches other statistics from the teacher model such as intermediate feature representations [Romero et al., 2014], Jacobian matrices [Srinivas and Fleuret, 2018], distributions [Huang and Wang, 2017], Gram matrices [Yim et al., 2017].”
Gradients as Features for Deep Representation Learning (2020, background): “…more recently for generating adversarial samples (Szegedy et al., 2013), crafting defense strategies (Goodfellow et al., 2015), facilitating knowledge distillation (Sinha et al., 2018; Srinivas & Fleuret, 2018), and boosting multi-task and meta learning (Sinha et al., 2018; Achille et al., 2019).”
Matching Guided Distillation (2020, background): “In the end, although the work of Jacobian matching-based distillation [31] seems related to ours, it still suffers from the problems discussed in Section.”
Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes (2023, methodology): “Although prior work has explored using data augmentation techniques to reduce this hunger for data (Tang et al., 2019; Liang et al., 2020; Srinivas and Fleuret, 2018; Milli et al., 2019), we propose an alternative approach: we reduce the need for large unlabeled data by distilling not just labels…”
A comprehensive survey on model compression and acceleration (2020, background): “In another study, Srinivas and Fleuret (2018) proposed the matching of Jacobian (the gradients) to transfer the knowledge from the teacher model to a student model.”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “Based on the definition of layer group [261], in which a group of layers have same spatial size, AT [260], FSP [254], Jacobian [202], MEAL [198], KSANC [27] and Kulkarni et al.”
Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation (2021, background): “…softened probabilities of the teacher network but also the hidden feature vector so that the student could be trained with rich information from the teacher [Romero et al. , 2014; Zagoruyko and Komodakis, 2016a; Srinivas and Fleuret, 2018; Kim et al. , 2018; Heo et al. , 2019b; Heo et al. , 2019a].”
Knowledge As Priors: Cross-Modal Knowledge Generalization for Datasets Without Superior Knowledge (2020, background): “They transfer knowledge across different modalities by a Teacher-Student scheme [16, 41, 54].”
Lipschitz Continuity Guided Knowledge Distillation (2021, background): “Setup Teacher Baseline KD FitNets AT Jacobian FT AB OFD AFD LONDON [16] [34] [48] [35] [21] [15] [14] [41] (ours) (a) 21.”
2DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Clouds (2022, methodology): “For instance, aligning attention maps [49] and Jacobean matrixes [50] were independently applied.”
A Comprehensive Overhaul of Feature Distillation (2019, methodology): “Therefore, in methods like AT [30], FSP [28], Jacobian [26] and FT [13], the same amount of information is lost in both the”
Ensemble Knowledge Distillation for Learning Improved and Efficient Networks (2019, methodology): “Other methods explored knowledge distillation using activation maps [Heo et al. , 2019], attention maps [Zagoruyko and Komodakis, 2016], Jacobians [Srinivas and Fleuret, 2018], and unsupervised feature factors [Kim et al. , 2018].”
R-DFCIL: Relation-Guided Representation Learning for Data-Free Class Incremental Learning (2022, methodology): “Since then, various KD methods [22,13,26,18] have been developed.”
Understanding and Improving Knowledge Distillation (2020, background): “…layer activations (Romero et al., 2014), auxiliary information (Vapnik & Izmailov, 2015), Jacobian matrix of the model parameters (Czarnecki et al., 2017; Srinivas & Fleuret, 2018), Gram matrix derived from pairs of layers (Yim et al., 2017), activation boundary (Heo et al., 2019), etc.”
Knowledge Distillation from Few Samples (2018, background): “…extended this approach by matching other statistics, including intermediate feature responses (Romero et al., 2015; Chen et al., 2016), gradient (Srinivas & Fleuret, 2018), distribution (Huang & Wang, 2017), Gram matrix (Yim et al., 2017), etc. Deep mutual learning (Zhang et al., 2018) trains a…”