On the Depth of Deep Neural Networks: A Theoretical View

By Shizhao Sun, Liwei Wang, Xiaoguang Liu, Tie-Yan Liu. 2015

176 citations recorded by the source corpus.

People believe that depth plays an important role in success of deep neural networks (DNN). However, this belief lacks solid theoretical justifications as far as we know. We investigate role of depth from perspective of margin bound. In margin bound, expected error is upper bounded by empirical margin error plus Rademacher Average (RA) based capacity term. First, we derive an upper bound for RA of DNN, and show that it increases with increasing depth. This indicates negative impact of depth on test performance. Second, we show that deeper networks tend to have larger representation power (measured by Betti numbers based complexity) than shallower networks in multi-class setting, and thus can lead to smaller empirical margin error. This implies positive impact of depth. The combination of these two results shows that for DNN with restricted number of hidden units, increasing depth is not always good since there is a tradeoff between positive and negative impacts. These results inspire us to seek alternative ways to achieve positive impact of depth, e.g., imposing margin-based penalty terms to cross entropy loss so as to reduce empirical margin error without increasing depth. Our experiments show that in this way, we achieve significantly better test performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Towards Better Analysis of Deep Convolutional Neural Networks (2016, neutral): “These findings are consistent with previous research [54].”
AdaNet: Adaptive Structural Learning of Artificial Neural Networks (2016, background): “It can also be shown to relate to other classical notions of model complexity, such as the VC-dimension and covering number (see e.g. [Vapnik, 1998, Mohri et al., 2012]).”
Understanding Generalization through Visualizations (2019, unknown): “…theory fails to explain generalization in over-parameterized neural nets, as the complexity of networks is often large (exponential in depth (Sun et al., 2016; Neyshabur et al., 2015; Xie et al., 2015) or linear in the number of parameters (Shalev-Shwartz and Ben-David, 2014; Bartlett et…”
How Does Mixup Help With Robustness and Generalization? (2020, background): “While there have been many enlightening studies trying to establish generalization theory for modern machine learning algorithms (Sun et al., 2015; Neyshabur et al., 2015; Hardt et al., 2016; Bartlett et al., 2017; Kawaguchi et al., 2017; Arora et al., 2018; Neyshabur & Li, 2019), few existing studi”
Artificial intelligence and machine learning approaches to energy demand-side response: A systematic review (2020, background): “14 not fully understood why they work so well in certain types of problems [219], and it should be noted that arbitrarily increasing the depth of an ANN might not always yield the best results [220].”
Deep learning generalizes because the parameter-function map is biased towards simple functions (2018, background): “[50] Shizhao Sun, Wei Chen, Liwei Wang, Xiaoguang Liu, and Tie-Yan Liu.”
On the Geometry of Adversarial Examples (2018, background): “There have been attempts to incorporate maximum margins into deep learning (Sun et al. (2016); Liu et al. (2016); Liang et al. (2017); Elsayed et al. (2018)), often by designing loss functions that encourage large margins at either the output (Sun et al. (2016)) or at any layer (Elsayed et al.…”