Sobolev Training for Neural Networks

By Wojciech M. Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Swirszcz, Razvan Pascanu. 2017

337 citations recorded by the source corpus.

At the heart of deep learning we aim to use neural networks as function approximators - training them to produce outputs from inputs in emulation of a ground truth function or data creation process. In many cases we only have access to input-output pairs from the ground truth, however it is becoming more common to have access to derivatives of the target output with respect to the input - for example when the ground truth function is itself a neural network such as in network compression or distillation. Generally these target derivatives are not computed, or are ignored. This paper introduces Sobolev Training for neural networks, which is a method for incorporating these target derivatives in addition the to target values while training. By optimising neural networks to not only approximate the function's outputs but also the function's derivatives we encode additional information about the target function within the parameters of the neural network. Thereby we can improve the quality of our predictors, as well as the data-efficiency and generalization capabilities of our learned function approximation. We provide theoretical justifications for such an approach as well as examples of empirical evidence on three distinct domains: regression on classical optimisation datasets, distilling policies of an agent playing Atari, and on large-scale applications of synthetic gradients. In all three domains the use of Sobolev Training, employing target derivatives in addition to target values, results in models with higher accuracy and stronger generalisation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

mixup: Beyond Empirical Risk Minimization (2017, methodology): “The linearity constraint, through its effect on the derivatives of the function approximated, also relates mixup to other methods such as Sobolev training of neural networks (Czarnecki et al., 2017) or WGAN-GP (Gulrajani et al.”
Born Again Neural Networks (2018, background): “Czarnecki et al. (2017) try to minimize the difference between teacher and student derivatives of the loss with respect to the input in addition to minimizing the divergence from teacher predictions.”
Improved Knowledge Distillation via Teacher Assistant (2019, background): “Czarnecki et al. (2017) minimized the difference between teacher and student derivatives of the loss combined with the divergence from teacher predictions while Tarvainen and Valpola (2017) uses averaging model weights instead of target predictions.”
Neural probabilistic motor primitives for humanoid control (2018, background): “Explicit Jacobian matching has been proposed elsewhere, for example in Czarnecki et al. (2017).”
Geometric deep learning for computational mechanics Part I: Anisotropic Hyperelasticity (2020, methodology): “Sobolev training we adopted from Czarnecki et al. (2017) is designed to maximize the utilization of data by leveraging the available additional higher order data in the form of higher order constraints in the training objective function.”
On Knowledge distillation from complex networks for response prediction (2019, background): “The so called Sobolev training (Czarnecki et al., 2017) exploits this information and adds two more extra terms to the objective in (1).”
Monge-Ampère Flow for Generative Modeling (2018, background): “Since one has the force information for both target problem and the network as well, one can introduce an additional term in the loss function for the force difference Czarnecki et al. (2017); Zhang et al. (2018a).”
Mix&Match - Agent Curricula for Reinforcement Learning (2018, methodology): “Furthermore, for speeding up convergence of distillation one could use Sobolev Training (Czarnecki et al., 2017) and match both policy and its Jacobian matrix.”
Knowledge Transfer with Jacobian Matching (2018, background): “Recent works (Czarnecki et al., 2017; Zagoruyko & Komodakis, 2017) sought to match the Jacobians of S and T .”
Gradient Adversarial Training of Neural Networks (2018, methodology): “A solution for student, S ( x ) which jointly minimizes the supervised loss and ∇ S ( x ) = ∇ T ( x ) exists, as proved in [4]. distributions owing GREAT uses a discriminator to match the gradient to the success of adversarial losses [7] over traditional regression-based loses.”
Improved Knowledge Distillation via Teacher Assistant: Bridging the Gap Between Student and Teacher (2019, background): “Czarnecki et al. (2017) minimized the difference between teacher and student derivatives of the loss combined with the divergence from teacher predictions.”
Understanding and Improving Knowledge Distillation (2020, methodology): “…investigated: layer activations (Romero et al., 2014), auxiliary information (Vapnik & Izmailov, 2015), Jacobian matrix of the model parameters (Czarnecki et al., 2017; Srinivas & Fleuret, 2018), Gram matrix derived from pairs of layers (Yim et al., 2017), activation boundary (Heo et al.,…”
Robust Learning with Jacobian Regularization (2019, background): “8 Different incarnations of a similar idea have appeared in the following decades [Simard et al., 1992, Mitchell and Thrun, 1993, Aires et al., 1999, Rifai et al., 2011, Gulrajani et al., 2017, Yoshida and Miyato, 2017, Czarnecki et al., 2017, Jakubovitz and Giryes, 2018].”