Why ResNet Works? Residuals Generalize

By Fengxiang He, Tongliang Liu, Dacheng Tao. 2019

373 citations recorded by the source corpus.

Residual connections significantly boost the performance of deep neural networks. However, few theoretical results address the influence of residuals on the hypothesis complexity and the generalization ability of deep neural networks. This article studies the influence of residual connections on the hypothesis complexity of the neural network in terms of the covering number of its hypothesis space. We first present an upper bound of the covering number of networks with residual connections. This bound shares a similar structure with that of neural networks without residual connections. This result suggests that moving a weight matrix or nonlinear activation from the bone to a vine would not increase the hypothesis space. Afterward, an $\mathcal O(1 / \sqrt {N})$ margin-based multiclass generalization bound is obtained for ResNet, as an exemplary case of any deep neural network with residual connections. Generalization guarantees for similar state-of-the-art neural network architectures, such as DenseNet and ResNeXt, are straightforward. According to the obtained generalization bound, we should introduce regularization terms to control the magnitude of the norms of weight matrices not to increase too much, in practice, to ensure a good generalization ability, which justifies the technique of weight decay.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Knowledge Distillation: A Survey (2020, methodology): “…are other special interesting applications of knowledge distillation, such as neural architecture search (Macko et al. 2019; Bashivanetal.2019),interpretabilityofdeepneuralnetworks (Liu et al. 2018b), and federated learning (Bistritz et al. 2020; Lin et al. 2020; Seo et al. 2020; He et al. 2020a).”
On Measuring Excess Capacity in Neural Networks (2022, background): “He et al. [19] investigate covering number bounds for function spaces as considered above.”