Orthogonal Deep Neural Networks

By Shuai Li, K. Jia, Tongliang Liu, Dacheng Tao. 2019

164 citations recorded by the source corpus.

In this paper, we introduce the algorithms of Orthogonal Deep Neural Networks (OrthDNNs) to connect with recent interest of spectrally regularized deep learning methods. OrthDNNs are theoretically motivated by generalization analysis of modern DNNs, with the aim to find solution properties of network weights that guarantee better generalization. To this end, we first prove that DNNs are of local isometry on data distributions of practical interest; by using a new covering of the sample space and introducing the local isometry property of DNNs into generalization analysis, we establish a new generalization error bound that is both scale- and range-sensitive to singular value spectrum of each of networks’ weight matrices. We prove that the optimal bound w.r.t. the degree of isometry is attained when each weight matrix has a spectrum of equal singular values, among which orthogonal weight matrix or a non-square one with orthonormal rows or columns is the most straightforward choice, suggesting the algorithms of OrthDNNs. We present both algorithms of strict and approximate OrthDNNs, and for the later ones we propose a simple yet effective algorithm called Singular Value Bounding (SVB), which performs as well as strict OrthDNNs, but at a much lower computational cost. We also propose Bounded Batch Normalization (BBN) to make compatible use of batch normalization with OrthDNNs. We conduct extensive comparative studies by using modern architectures on benchmark image classification. Experiments show the efficacy of OrthDNNs.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

1. Hui Tang
2. Ling Shao
3. K. Jia
4. Li Liu
5. Lei Huang

In Computer Vision and Pattern Recognition

1. Ling Shao
2. Li Liu
3. Fan Zhu
4. K. Jia
5. Yi Zhou
6. Hui Tang
7. Jie Qin
8. Lei Huang

In Media Technology

1. Hui Tang
2. Ling Shao
3. Fan Zhu
4. Yi Zhou
5. K. Jia
6. Li Liu
7. Lei Huang

Citations in the corpus

• Continual Learning in Low-rank Orthogonal Subspaces (2020, neutral): “Whereas optimizing over the Stiefel manifold ensure strict orthogonality in the weights, [Jia et al., 2019] proposed an algorithm, Singular Value Bounding (SVB), for soft orthogonality constraints.”
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, methodology): “…on weights is orthogonality, which is represented as Orthogonality was first used in the square hidden-to-hidden weight matrices of RNNs [143], [144], [145], [146], [147], [148], [149], and then further extended to the more general rectangular matrices in DNNs [52], [150], [151], [152], [153].”
• Unsupervised Domain Adaptation via Structurally Regularized Deep Clustering (2020, methodology): “We parameterize the feature embedding function φ(·;θ) and classifier f(·;θ) as a deep network [21, 25, 26, 65], (a) (b) (c)”