Double backpropagation increasing generalization performance

By Harris Drucker, Y. Le Cun. 1991

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Adversarial Vulnerability of Neural Networks Increases With Input Dimension (2018, background): “Since the trendsetting paper of Goodfellow et al. (2014), Convolutional Neural Networks (CNNs) have been found utterly vulnerable to adversarial examples: an adversary can drive the performance of state-of-the art CNNs down to chance-level with imperceptible changes of the inputs. A number of studie”
Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulation (2017, background): “It is interesting to note that classical regularization functionals which penalize derivatives of the resulting classifier function are not typically used in deep learning, but see [6, 11].”
Gradient Adversarial Training of Neural Networks (2018, methodology): “The information from gradient tensors have been employed classically for regularization [5] and more recently for visualizing saliency maps [28], interpreting DNNs [32, 29], generating adversarial examples [8] and weakly supervised object localization[27].”
Interpreting Adversarial Robustness: A View from Decision Surface in Input Space (2018, background): “To solve this problem, double backpropagation (Drucker & Le Cun (1991)) is needed.”
On the Effectiveness of Mitigating Data Poisoning Attacks with Gradient Shaping (2020, background): “Gradient Regularization as a Defense Prior work, especially in the context of neural networks, has proposed imposing regularization penalties on a model’s gradients to improve the accuracy [16], interpretability [33] or robustness against adverarial examples [27].”
Learning Energy-Based Models in High-Dimensional Spaces with Multiscale Denoising-Score Matching (2019, background): “With these premises, one can visualize the conﬁguration of noisy and noiseless data points that enter the learning process: A data point xxx sampled from X and its noisy version ˜ xxx always lie on a line which is almost perpendicular to the tangent space T xxx X and intersects X at xxx .”
First-Order Adversarial Vulnerability of Neural Networks and Input Dimension (2018, background): “Incidentally, for ‖·‖ = ‖·‖2, this new loss reduces to an old regularizationscheme proposed by Drucker & LeCun (1991) called doublebackpropagation.”
A Kernel Perspective for Regularizing Deep Neural Networks (2018, unknown): “…to adversarial robustness (Cisse et al., 2017; Madry et al., 2018; Simon-Gabriel et al., 2019; Roth et al., 2018), to improving generalization (Drucker & Le Cun, 1991; Miyato et al., 2018b; Sedghi et al., 2019; Simard et al., 1998; Yoshida & Miyato, 2017), and stable training of generative…”
Robust Learning with Jacobian Regularization (2019, unknown): “To our knowledge, double backpropagation [Drucker and LeCun, 1991, 1992] is the earliest attempt to penalize large derivatives with respect to input data, in which ( ∂ L super /∂ x ) 2 is added to the loss in order to reduce the generalization gap.”
Learning Smooth Neural Functions via Lipschitz Regularization (2022, methodology): “Then we can encourage smoothness at these input samples by penalizing the norm of the Jacobian [Drucker and Le Cun 1991; Gulrajani et al. 2017; Hoffman et al. 2019; Jakubovitz and Giryes 2018; Varga et al. 2017] or the Hessian [Moosavi-Dezfooli et al. 2019].”
Attentive Single-Tasking of Multiple Tasks (2019, methodology): “This loss is minimized during training through double back-propagation [10], and leads to an automatic balancing of loss terms, while promoting compartmentalization between task-speciﬁc and shared blocks.”