Improving generalization performance using double backpropagation

By Harris Drucker, Yann LeCun. 1992

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

mixup: Beyond Empirical Risk Minimization (2017, methodology): “Among the several methods aiming to solve this problem, some have proposed to penalize the norm of the Jacobian of the model to control its Lipschitz constant (Drucker & Le Cun, 1992; Cisse et al., 2017; Bartlett et al., 2017; Hein & Andriushchenko, 2017).”
A Style-Based Generator Architecture for Generative Adversarial Networks (2018, methodology): “FFHQ uses WGAN-GP for configuration a and nonsaturating loss [21] with R1 regularization [40, 47, 13] for configurations b–f.”
Resolution-robust Large Mask Inpainting with Fourier Convolutions (2021, methodology): “In the final loss we also use [31, 38, 7], and a discriminator-based perceptual loss or so-called feature matching loss—a perceptual loss on the features of discriminator network L DiscPL [45].”
Gradient descent GAN optimization is locally stable (2017, methodology): “Speciﬁcally, we propose the regularized update Although these update equations do require that we differentiate with respect to a function of another gradient term, such “double backprop” terms (see e.g., [Drucker and Le Cun, 1992]) are easily computed by modern automatic differentiation tools.”
Distillation as a Defense to Adversarial Perturbations Against Deep Neural Networks (2015, methodology): “, using double backpropagation [31]) only provides constraints very near training examples, so it does not solve the adversarial perturbation problem.”
Occupancy Networks: Learning 3D Reconstruction in Function Space (2018, methodology): “com/sp4cerat/Fast-Quadric-Mesh-Simplification in (6) uses second order gradient information and can be efficiently implemented using Double-Backpropagation [15].”
Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer (2016, methodology): “Median of 5 runs test errors are reported. norm type error baseline (no attention transfer) 13.5 min-l2 Drucker & LeCun (1992) 12.5 grad-based AT 12.1 KD 12.1 symmetry norm 11.8 activation-based AT 11.2 Table 3: Performance of various gradient-based attention methods on CIFAR-10.”
Parseval Networks: Improving Robustness to Adversarial Examples (2017, methodology): “Previous work, such as double backpropagation (Drucker & Le Cun, 1992) has also explored jacobian normalization as a way to improve generalization.”
Simple and Scalable Epistemic Uncertainty Estimation Using a Single Deep Deterministic Neural Network (2020, methodology): “Regularising the Jacobian has a long history, starting with Drucker & Le Cun (1992) and more recently Ross & Doshi-Velez (2018).”
Sorting out Lipschitz function approximation (2018, methodology): “Other regularization techniques penalize the Jacobian of the network, thereby constraining the Lipschitz constant locally around the data [21, 15, 44].”
Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients (2017, background): “It was ﬁrst introduced by Drucker and Le Cun (1992) as “double back-propagation”, which trains neural networks by minimizing not just the “energy” of the network but the rate of change of that energy with respect to the input features.”
Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey (2018, methodology): “Ross and Doshi-Velez [52] studied input gradient regularization [165] as a method for adversarial robustness.”
Adversarial Latent Autoencoders (2020, methodology): “In addition, we use gradient regularization techniques [8, 36, 43].”
Learning Explainable Models Using Attribution Priors (2019, methodology): “Graph convolutional networks: We followed the implementation of graph convolution described in Kipf and Welling [7].”
The Singular Values of Convolutional Layers (2018, background): “These considerations have led authors to regularize networks by driving down the operator norms of network layers (Drucker and Le Cun, 1992; Hein and Andriushchenko, 2017; Yoshida and Miyato, 2017; Miyato et al., 2018).”