Spectral Norm Regularization for Improving the Generalizability of Deep Learning

By Yuichi Yoshida, Takeru Miyato. 2017

415 citations recorded by the source corpus.

We investigate the generalizability of deep learning based on the sensitivity to input perturbation. We hypothesize that the high sensitivity to the perturbation of data degrades the performance on it. To reduce the sensitivity to perturbation, we propose a simple and effective regularization method, referred to as spectral norm regularization, which penalizes the high spectral norm of weight matrices in neural networks. We provide supportive evidence for the abovementioned hypothesis by experimentally confirming that the models trained using spectral norm regularization exhibit better generalizability than other baseline methods.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Spectral Normalization for Generative Adversarial Networks (2018, methodology): “A method we would introduce in this paper, called spectral normalization , is a method that aims to skirt this issue by normalizing the weight matrices using the technique devised by Yoshida & Miyato (2017).”
PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization (2019, methodology): “Using a low-rank update (as we do) can be viewed as implicitly performing spectral regularization (Gunasekar et al., 2018) and hence can be expected to have good generalization properties (Yoshida & Miyato, 2017).”
Diversity With Cooperation: Ensemble Methods for Few-Shot Classification (2019, methodology): “Cooperation encourages the non-ground truth probabilities (in red) to be similar, after normalization, whereas diversity encourages orthogonality. deep networks [2, 35], we choose in this paper to focus on variance-reduction principles based on ensemble methods.”
On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning (2022, background): “As stated in [86, 67], an effective way to bound the Lipschitz constant of f(·) is to compute the constants of each linear projection H and then aggregate them.”
Sorting out Lipschitz function approximation (2018, methodology): “The intuition for this algorithm follows that of Yoshida & Miyato [56], who apply power iteration to estimate the singular values of the weight matrices online.”
Regularisation of neural networks by enforcing Lipschitz continuity (2018, methodology): “Yoshida and Miyato (2017) proposed a new regularisation scheme in the form of a term in the loss function that penalises the sum of spectral norms of the weight matrices.”
Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks (2018, methodology): “At training time, we need only one iteration of the above update formula as with Yoshida and Miyato [33].”
The Singular Values of Convolutional Layers (2018, methodology): “Up until now, authors seeking to control the operator norm of convolutional layers have resorted to approximations (Yoshida and Miyato, 2017; Miyato et al., 2018; Gouk et al., 2018a).”
Heavy-Tailed Universality Predicts Trends in Test Accuracies for Very Large Pre-Trained Deep Neural Networks (2019, background): “, [2, 3, 4] and [5, 6, 7, 8, 9, 10, 11, 12, 13, 14], as well as much older [15, 16].”
Lipschitz Networks and Distributional Robustness (2018, methodology): “Lipschitz-constrained networks have been shown to train faster, and have better robustness and generalisation properties (Cisse et al., 2017; Yoshida and Miyato, 2017; Miyato et al., 2018; Gouk et al., 2018).”
A Kernel Perspective for Regularizing Deep Neural Networks (2018, positive): “Depending on the chosen approximation, we recover then many existing principles such as spectral norm regularization (Cisse et al., 2017; Yoshida & Miyato, 2017; Miyato et al., 2018a; Sedghi et al., 2019), gradient penalties and double backpropagation (Drucker & Le Cun, 1991; Simon-Gabriel et al.,…”
Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning (2018, methodology): “This includes: work on large-batch learning and the generalization gap [148, 72, 64, 57, 67, 131, 66, 146, 94, 151, 152]; work on Energy Landscape approaches to NN training [70, 149, 44, 123, 30, 29, 27, 65, 47, 12, 104, 150, 50, 83, 82, 95]; work on using weight matrices or properties of weight mat”
L2-Nonexpansive Neural Networks (2018, methodology): “Yoshida & Miyato (2017) proposes a method to regularize the spectral radius of weight matrices and shows its effect in reducing generalization gap.”
Robust Learning with Jacobian Regularization (2019, background): “8 Different incarnations of a similar idea have appeared in the following decades [Simard et al., 1992, Mitchell and Thrun, 1993, Aires et al., 1999, Rifai et al., 2011, Gulrajani et al., 2017, Yoshida and Miyato, 2017, Czarnecki et al., 2017, Jakubovitz and Giryes, 2018].”
Generative Adversarial Networks: A Survey and Taxonomy (2019, background): “The results suggest that the original GAN [1] with spectral normalization [67] is a good starting choice when applying GANs to a new dataset.”
Learning Smooth Neural Functions via Lipschitz Regularization (2022, methodology): “4, we use the spectral norm for the method by Yoshida and Miyato [2017] and show that the same 𝛼 results in different behavior for networks with different capacities.”
Lipschitz Continuity Guided Knowledge Distillation (2021, methodology): “Apart from some theoretical studies [1, 27, 30] explaining that novel generalization bounds critically rely on the Lipschitz constant of the neural network, Lipschitz continuity of neural networks is widely studied for achieving the state-of-the-art performance in many deep learning topics: (i) In i”
On Regularization and Robustness of Deep Neural Networks (2018, methodology): “We consider the method of Yoshida & Miyato (2017), which computes such singular vectors approximately using one or two iterations of the power method, as well as a more costly approach using the full SVD. Constraining the spectral norms with a continuation approach.”