Improving Training of Deep Neural Networks via Singular Value Bounding

By K. Jia, Dacheng Tao, Shenghua Gao, Xiangmin Xu. 2016

97 citations recorded by the source corpus.

Deep learning methods achieve great success recently on many computer vision problems. In spite of these practical successes, optimization of deep networks remains an active topic in deep learning research. In this work, we focus on investigation of the network solution properties that can potentially lead to good performance. Our research is inspired by theoretical and empirical results that use orthogonal matrices to initialize networks, but we are interested in investigating how orthogonal weight matrices perform when network training converges. To this end, we propose to constrain the solutions of weight matrices in the orthogonal feasible set during the whole process of network training, and achieve this by a simple yet effective method called Singular Value Bounding (SVB). In SVB, all singular values of each weight matrix are simply bounded in a narrow band around the value of 1. Based on the same motivation, we also propose Bounded Batch Normalization (BBN), which improves Batch Normalization by removing its potential risk of ill-conditioned layer transform. We present both theoretical and empirical results to justify our proposed methods. Experiments on benchmark image classification datasets show the efficacy of our proposed SVB and BBN. In particular, we achieve the state-of-the-art results of 3.06% error rate on CIFAR10 and 16.90% on CIFAR100, using off-the-shelf network architectures (Wide ResNets). Our preliminary results on ImageNet also show the promise in large-scale learning. We release the implementation code of our methods at www.aperture-lab.net/research/svb.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Mingkui Tan
3. Yong Guo
4. Jian Chen
5. Lei Zhang
6. K. Jia
7. Tongliang Liu
8. Anton van den Hengel

In Computational Mechanics

1. Kai Zhang
2. Wangmeng Zuo
3. Lei Zhang
4. K. Jia
5. Dacheng Tao
6. Tongliang Liu
7. Yong Guo
8. Jian Chen

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Lei Zhang
4. Wangmeng Zuo
5. Ming-Hsuan Yang
6. Kai Zhang
7. Radu Timofte
8. Chunhua Shen

In Optimization & Training

1. Tongliang Liu
2. Yong Guo
3. Dacheng Tao
4. Kai Zhang
5. Wangmeng Zuo
6. Jian Chen
7. Shuai Li
8. K. Jia

Citations in the corpus

• Orthogonal Deep Neural Networks (2019, methodology): “These methods impose explicit regularization on weight matrices of network layers by penalizing either their spectral norms [50], [55] or the whole spectrums of their singular values [4], [12], [29], [57].”
• Analysis of Invariance and Robustness via Invertibility of ReLU-Networks (2018, background): “While many defense strategies like [Cisse et al., 2017] or [Jia, 2017] focus on the largest singular value, the behavior of the smaller singular values are of special interest for the stability of the inverse.”
• The Shallow End: Empowering Shallower Deep-Convolutional Networks through Auxiliary Outputs (2016, methodology): “Pool [3], Highway Network [22], StochResNet [8], ResNet [6], Wide Residual Network(Wide ResNet) [30], ResNet of ResNet [33], DenseNet [7] and SVB-Wide ResNet [9].”
• Orthogonal Jacobian Regularization for Unsupervised Disentanglement in Image Generation (2021, background): “Many recent studies have been given to incorporate the orthogonality for improving deep network training [5, 19, 31, 36, 43, 44].”
• FFDNet: Toward a Fast and Flexible Solution for CNN-Based Image Denoising (2017, methodology): “According to our experiments, we empirically find that the orthogonal initialization of the convolution filters [43], [46] works well in suppressing the above mentioned visual artifacts.”