Regularisation of neural networks by enforcing Lipschitz continuity

By Henry Gouk, Eibe Frank, Bernhard Pfahringer, Michael J. Cree. 2018

626 citations recorded by the source corpus.

We investigate the effect of explicitly enforcing the Lipschitz continuity of neural networks with respect to their inputs. To this end, we provide a simple technique for computing an upper bound to the Lipschitz constant—for multiple p-norms—of a feed forward neural network composed of commonly used layer types. Our technique is then used to formulate training a neural network with a bounded Lipschitz constant as a constrained optimisation problem that can be solved using projected stochastic gradient methods. Our evaluation study shows that the performance of the resulting models exceeds that of models trained with other common regularisers. We also provide evidence that the hyperparameters are intuitive to tune, demonstrate how the choice of norm for computing the Lipschitz constant impacts the resulting model, and show that the performance gains provided by our method are particularly noticeable when only a small amount of training data is available.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

DeepMDP: Learning Continuous Latent Space Models for Representation Learning (2019, methodology): “Lipschitz continuity has also been proposed as a regularization method by Gouk et al. (2018), who provided an approach to compute an upper bound to the Lipschitz constant of neural nets.”
Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness (2020, methodology): “As shown, for every minibatch step, the model first updates the hidden-layer weights {Wl ,bl} l=1 and the trainable output weights β = {βk}k=1 via SGD, then performs spectral normalization using power iteration method ([26] which has time complexity O(∑L−1 l=1 Dl)), and finally performs precision ma”
Certified Adversarial Robustness via Randomized Smoothing (2019, methodology): “Some conservative methods bound the global Lipschitz constant of the neural network (Gouk et al., 2018; Tsuzuku et al., 2018; Anil et al., 2019; Cisse et al., 2017), but these approaches tend to be very loose on expressive networks.”
Theoretical evidence for adversarial robustness through randomization: the case of the Exponential family (2019, background): “It is well known that in the case of deterministic neural networks, the Lipschitz constant becomes bigger as the number of layers increases [19].”
On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning (2022, methodology): “The computation of ||Wk||SN can be done [55, 29] naively through the Singular Value Decomposition (SVD), yielding, among the others, the largest singular value.”
Invertible Residual Networks (2018, methodology): “Unlike [19], we directly estimate the spectral norm of W i by performing power-iteration using W i and W Ti as proposed in [12].”
Sorting out Lipschitz function approximation (2018, methodology): “Several methods have been proposed to train Lipschitz neural networks [12, 56, 33, 20].”
Normalizing Flows for Probabilistic Modeling and Inference (2019, background): “Many neural-network layers meet these requirements (Gouk et al., 2018), and unlike the neural architectures described in section 3 that require careful structural assumptions to ensure invertibility and tractability of their Jacobian determinant, gφ has no such requirements.”
Residual Flows for Invertible Generative Modeling (2019, methodology): “We used spectral normalization for convolutions (Gouk et al., 2018).”
The Singular Values of Convolutional Layers (2018, methodology): “Up until now, authors seeking to control the operator norm of convolutional layers have resorted to approximations (Yoshida and Miyato, 2017; Miyato et al., 2018; Gouk et al., 2018a).”
AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Time (2022, background): “where Ls is the Lipschitz constant of the normalization function [58], [59].”
Avoiding Resentment Via Monotonic Fairness (2019, background): “continuous functions, although the Lipschitz constant may be large [38, 20, 3].”
Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks (2019, background): “Finally, an efficient computation of these bounds can be useful in either assessing robustness after training [29, 30, 13] or promoting robustness during training [40, 36, 17].”
Residual Networks as Flows of Diffeomorphisms (2019, background): “This property guarantees that the learned residual units are Lipschitz continuous, which is related to recent works investigating explicit constraints on Lipschitz continuity in neural networks [11].”
Relaxing Bijectivity Constraints with Continuously Indexed Normalising Flows (2019, methodology): “The Lipschitz constraint is enforced by spectral normalisation (Miyato et al., 2018; Gouk et al., 2018) and ensures each f` is bijective.”
Coarse graining molecular dynamics with graph neural networks. (2020, methodology): “Third, CGSchNet employs less regularization; particularly, it does not require the extra step of enforcing a Lipschitz constraint 37 on its network’s weight matrices as was found to be necessary for CGnet 20 .”
Plug-and-Play Methods Provably Converge with Properly Trained Denoisers (2019, methodology): “Provided that we use 1 -Lipschitz nonlinearities (such as ReLU), the Lipschitz constant of a layer is upper-bounded by the spectral norm of its weight, and the Lipschitz constant of the full network is bounded by the product of spectral norms of all layers (Gouk et al., 2018).”
Hausdorff GAN: Improving GAN Generation Quality With Hausdorff Metric (2021, background): “However, such a strategy does not completely correspond to penalizing the spectral norm [15], which causes the generated data to be mapped into a speciﬁc subspace in color space, rendering all generated objects to take the color from that subspace.”