The Singular Values of Convolutional Layers

By Hanie Sedghi, Vineet Gupta, Philip M. Long. 2018

225 citations recorded by the source corpus.

We characterize the singular values of the linear transformation associated with a standard 2D multi-channel convolutional layer, enabling their efficient computation. This characterization also leads to an algorithm for projecting a convolutional layer onto an operator-norm ball. We show that this is an effective regularizer; for example, it improves the test error of a deep residual network using batch normalization on CIFAR-10 from 6.2\% to 5.3\%.

Subfields: Atomic and Molecular Physics, and Optics, Computational Mechanics, Computational Theory and Mathematics, Model Architectures, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Atomic and Molecular Physics, and Optics

In Computational Mechanics

In Computational Theory and Mathematics

In Model Architectures

In Scaling & Learning Theory

Citations in the corpus

What is being transferred in transfer learning? (2020, methodology): “Appendix C: Spectrum of weight matrices We recover the spectrum of every module using the algorithm derived in [39] and we also look at the spectrum for the whole network in different cases of training.”
Invertible Residual Networks (2018, methodology): “Thus, after training we compute (cid:107) W i (cid:107) 2 exactly using the SVD on the Fourier transformed parameter matrix following [21] to show Lip( g ) 1 holds in all cases.”
Sorting out Lipschitz function approximation (2018, background): “We do not devote space to computing these constants here but instead point readers to other resources which address this question [20, 12, 40].”
Fantastic Generalization Measures and Where to Find Them (2019, methodology): “For convolutional operators, we compute the true singular value with the method proposed by Sedghi et al. (2018) through FFT.”
Improving MMD-GAN Training with Repulsive Loss Function (2018, methodology): “In addition, Sedghi et al. (2019) proposes to compute the exact singular values of convolution kernels using FFT and SVD.”
A Kernel Perspective for Regularizing Deep Neural Networks (2018, unknown): “, 2018), to improving generalization (Drucker & Le Cun, 1991; Miyato et al., 2018b; Sedghi et al., 2019; Simard et al., 1998; Yoshida & Miyato, 2017), and stable training of generative adversarial networks (Roth et al.”
Layer-adaptive Sparsity for the Magnitude-based Pruning (2020, background): “…fact that (cid:107) A (cid:107) ≤ (cid:107) A (cid:107) F holds for any matrix A 6 (where (cid:107) · (cid:107) F denotes the Frobenius norm), the optimization (5) can be relaxed to the Frobenius 4 An extension to convolutional neural network is straightforward; see, e.g., (Sedghi et al., 2019).”
Generalization bounds for deep convolutional neural networks (2019, methodology): “Since ||K −K0||σ ≤ || vec(K) − vec(K0)||1 (Sedghi et al., 2018), Theorem 2.”
Deep Networks from the Principle of Rate Reduction (2020, background): “(72) The matrix circ(z) is known as the doubly block circulant matrix associated with z (see, e.g., Abidi et al. (2016); Sedghi et al. (2018)).”
On Measuring Excess Capacity in Neural Networks (2022, methodology): “To implement the orthogonal projections onto C 1 and C 2 , we rely on work by Liu et al. [30] and Sedghi et al. [40], respectively.”
Learning Deep Kernels for Non-Parametric Two-Sample Tests (2020, background): “This is given in terms of the operator norm of various discrete Fourier transforms of the kernel matrix by Lemma 2 of Bibi et al. (2019); see also Theorem 6 of Sedghi et al. (2019).”
The intriguing role of module criticality in the generalization of deep networks (2019, methodology): “We calculate the spectrum of convolutional layers using the algorithm by Sedghi et al. (2019).”
Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theory (2019, methodology): “We employ the regularizer introduced in Sedghi et al. (2018) to create the protocols RankMin, to ﬁnd low-rank parameters, and RankMax, to ﬁnd high-rank parameters.”
On Regularization and Robustness of Deep Neural Networks (2018, methodology): “…2017; Madry et al., 2018; Simon-Gabriel et al., 2018; Roth et al., 2018), to improving generalization (Drucker & Le Cun, 1991; Miyato et al., 2018b; Sedghi et al., 2018; Simard et al., 1998; Yoshida & Miyato, 2017), and stable training of generative adversarial networks (Roth et al., 2017;…”
Globally-Robust Neural Networks (2021, background): “For more complicated networks, there is a growing body of work on computing layer-wise Lipschitz bounds for various types of layers that are commonly used in neural networks (Zou et al., 2019; Fazlyab et al., 2019; Sedghi et al., 2019; Singla & Feizi, 2019; Miyato et al., 2018).”
Size-free generalization bounds for convolutional neural networks (2019, methodology): “Since ||K −K0||σ ≤ || vec(K)− vec(K0)||1 [Sedghi et al., 2018], Theorem 2.1 yields the same bounds as a corollary if the definition of Fβ is replaced with the analogous definition using || vec(K)− vec(K0)||1.”
Few-Shot Adaptation of Generative Adversarial Networks (2020, methodology): “SVD is also used for a wide number of other applications, including regularization (Sedghi et al., 2018), and quantiﬁcation of entanglement (Martyn et al., 2020), and has also been used to build theoretical background for semantic development in neural networks (Saxe et al., 2019).”
Orthogonal Deep Neural Networks (2019, background): “For the lth layer, its exact weight matrix is in fact a function of the kernel tensor and contains doubly block circulant submatrices [48].”