Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks

By Lei Huang, Xianglong Liu, Bo Lang, Adams Wei Yu, Bo Li. 2017

264 citations recorded by the source corpus.

Orthogonal matrix has shown advantages in training Recurrent Neural Networks (RNNs), but such matrix is limited to be square for the hidden-to-hidden transformation in RNNs. In this paper, we generalize such square orthogonal matrix to orthogonal rectangular matrix and formulating this problem in feed-forward Neural Networks (FNNs) as Optimization over Multiple Dependent Stiefel Manifolds (OMDSM). We show that the orthogonal rectangular matrix can stabilize the distribution of network activations and regularize FNNs. We propose a novel orthogonal weight normalization method to solve OMDSM. Particularly, it constructs orthogonal transformation over proxy parameters to ensure the weight matrix is orthogonal. To guarantee stability, we minimize the distortions between proxy parameters and canonical weights over all tractable orthogonal transformations. In addition, we design orthogonal linear module (OLM) to learn orthogonal filter banks in practice, which can be used as an alternative to standard linear module. Extensive experiments demonstrate that by simply substituting OLM for standard linear module without revising any experimental protocols, our method improves the performance of the state-of-the-art networks, including Inception and residual networks on CIFAR and ImageNet datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Concept whitening for interpretable image recognition (2020, methodology): “Other work explores ways to learn orthogonal weights or representations for all types of neural networks (not just RNNs) (Harandi and Fernando, 2016; Huang et al., 2018a; Lezcano-Casado and Mart´ınez-Rubio, 2019; Lezama et al., 2018).”
Continual Learning in Low-rank Orthogonal Subspaces (2020, neutral): “While the weight matrices are assumed to be square in the earlier works, works including [Huang et al., 2018] considered learning non-square orthogonal matrices (called orthonormal matrices in this work) by optimizing over the Stiefel manifolds.”
Decorrelated Batch Normalization (2018, methodology): “A similar phenomenon is also observed in [22].”
Regularizing Neural Networks via Minimizing Hyperspherical Energy (2019, background): “…of prevailing meth-ods fall into this category, such as batch normalization [4], dropout [5], weight normalization [6], etc. Explicit regular-izations [7, 8, 9, 10, 11, 12] usually introduce some penalty terms for neuron weights, and jointly optimize them along with the other objective functions.”
Learning with Hyperspherical Uniformity (2021, background): “A popular choice out of many existing relational regularizations is the orthogonality regularization [13, 52, 4, 32, 33, 63].”
Learning Smooth Neural Functions via Lipschitz Regularization (2022, methodology): “Ourweight normalization shares the same spirit as the other weight normalization methods for accelerating training [Salimans and Kingma 2016] and generalization ability [Huang et al. 2018].”
Cascaded Projection: End-To-End Network Compression and Acceleration (2019, methodology): “[23] impose orthogonality constraints on the weights of a neural network during training using a method for backpropagation of gradients through structured linear algebra layers developed in [26, 27].”
Orthogonal Over-Parameterized Training (2020, methodology): “[27, 42, 3, 75, 58, 31] use principled orthogonalization methods to guarantee the neurons are orthogonal to each other.”
Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, methodology): “For example, a weight matrix with an orthogonality constraint (Eqn.31) is a real Stiefel manifold [52], [160].”
Iterative Normalization: Beyond Standardization Towards Efficient Whitening (2019, background): “, either through re-parameterization [36, 16, 15] or weight regularization [23, 32, 35]), and that specially design either scaling coefficients & bias values [1] or nonlinear function [20], to normalize activation implicitly [39].”
R-Drop: Regularized Dropout for Neural Networks (2021, background): “To tackle with over-ﬁtting, many regularization techniques have been proposed, e.g., weight decay [33, 31, 28, 67], dropout [24, 64, 4, 66, 61], normalization [27, 57, 3, 26, 71], adding noise [25, 52], layer-wise pre-training and initialization [12, 21], label-smoothing [62], and so on.”
Backward Feature Correction: How Deep Learning Performs Deep Learning (2020, unknown): “[39]).”
ABD-Net: Attentive but Diverse Person Re-Identification (2019, methodology): “l proposed in [11,12]. 2.3. Diversity via Orthogonality Orthogonality has been widely explored in deep learning to encourage the learning of informative and diverse features. In CNNs, several studies [34,35,36,37] perform regularization using “hard orthogonality constraints”, which typically depend ”