On Regularization and Robustness of Deep Neural Networks

By Alberto Bietti, Grégoire Mialon, Julien Mairal. 2018

15 citations recorded by the source corpus.

In this work, we study the connection between regularization and robustness of deep neural networks by viewing them as elements of a reproducing kernel Hilbert space (RKHS) of functions and by regularizing them using the RKHS norm. Even though this norm cannot be computed, we consider various approximations based on upper and lower bounds. These approximations lead to new strategies for regularization, but also to existing ones such as spectral norm penalties or constraints, gradient penalties, or adversarial training. Besides, the kernel framework allows us to obtain margin-based bounds on adversarial generalization. We show that our new algorithms lead to empirical benefits for learning on small datasets and learning adversarially robust models. We also discuss implications of our regularization framework for learning implicit generative models.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Parametric Noise Injection: Trainable Randomness to Improve Deep Neural Network Robustness Against Adversarial Attack (2018, background): “Recently, different works [6] have viewed the problem of adversarial examples from a unified perspective of model robustness and regularization.”
Group Invariance, Stability to Deformations, and Complexity of Deep Convolutional Representations (2017, background): “…to small, additive perturbations known as “adversarial examples” (Szegedy et al., 2014), which suggests that the RKHS norm of these models may be quite large, and that controlling it explicitly during learning might be important to learn more stable models (Bietti et al., 2018; Cisse et al., 2017).”