Multi-level Residual Networks from Dynamical Systems View

By Bo-Hao Chang, Lili Meng, Eldad Haber, Frederick Tung, David Begert. 2017

184 citations recorded by the source corpus.

Deep residual networks (ResNets) and their variants are widely used in many computer vision applications and natural language processing tasks. However, the theoretical principles for designing and training ResNets are still not fully understood. Recently, several points of view have emerged to try to interpret ResNet theoretically, such as unraveled view, unrolled iterative estimation and dynamical systems view. In this paper, we adopt the dynamical systems point of view, and analyze the lesioning properties of ResNet both theoretically and experimentally. Based on these analyses, we additionally propose a novel method for accelerating ResNet training. We apply the proposed method to train ResNets and Wide ResNets for three image classification benchmarks, reducing training time by more than 40\% with superior or on-par accuracy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Learning Deep Transformer Models for Machine Translation (2019, background): “In principle, residual networks can be seen as instances of the ordinary differential equation (ODE), behaving like the forward Euler discretization with an initial value (Chang et al., 2018; Chen et al., 2018b).”
Neural Ordinary Differential Equations (2018, neutral): “Finally, we derived an instantaneous version of the change of variables formula, and developed continuous-time normalizing ﬂ ows, which can scale to large layer sizes.”
Multistep Neural Networks for Data-driven Discovery of Nonlinear Dynamical Systems (2018, background): “…directions include the application of convolutional architectures [14] for mitigating the complexity associated with very high-dimensional inputs, as well as studying possible connections with recent studies linking deep neural networks with numerical methods and dynamical systems [30, 31].”
PDE-Net: Learning PDEs from Data (2017, background): “, 2016a;b), have close relationship with PDEs (Chen et al., 2015; E, 2017; Haber & Ruthotto, 2017; Sonoda & Murata, 2017; Lu et al., 2018; Chang et al., 2018).”
Learning Differential Equations that are Easy to Solve (2020, methodology): “Solving the resulting IVP is analogous to evaluating a very deep residual network in which the number of layers corresponds to the number of function evaluations of the solver (Chang et al., 2017; Ruthotto & Haber, 2018; Chen et al., 2018).”
Deep Neural Networks Motivated by Partial Differential Equations (2018, background): “ accuracy of around 85% in [6]. Additional Results. We refer to our previous works on multiscale learning [18], memory-ecient training [6], and a more extensive empirical study using other data sets [7]. 5 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000 4,500 5,000 !" number of training images &q”
CLIP-Q: Deep Network Compression Learning by In-parallel Pruning-Quantization (2018, background): “Complementary lines of work focus on accelerating deep network inference at test time [9, 10, 11, 30], or reducing deep network training times [1, 24].”
NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations (2018, methodology): “Moreover, stability problems in both the forward as well the backward path of VDNNs have been addressed by relying on well-known analytical approaches for continuous-time ODEs [15, 5].”
Continuous-in-Depth Neural Networks (2020, methodology): “This contrasts sharply with the refinements that are implemented for models that are not continuous-in-depth [9].”
Functional Gradient Boosting based on Residual Network Perception (2018, background): “Moreover, several studies (Weinan, 2017; Haber et al., 2017; Chang et al., 2017a;b; Lu et al., 2017) have pointed out that ResNet layers can be regarded as discretization steps of ordinary differential equations.”
PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid Deep Network (2018, methodology): “An interesting fact is that some popular networks in computer vision, such as ResNet[15, 16], have close relationship with ODEs/PDEs and can be naturally merged with traditional computational mathematics in various tasks [17, 18, 19, 20, 21, 22, 23, 24, 25, 26].”
On the Transformer Growth for Progressive BERT Training (2020, background): “For example, multi-level residual network (Chang et al., 2018) explores the possibility of augmenting network depth in a dynamic system of view and transforms each layer into two subsequent layers.”
Scalable Gradients for Stochastic Differential Equations (2020, methodology): “[11] used this method to compute the gradient with respect to parameters of a neural ODE, which is a particular model among many others inspired by the theory of dynamical systems [9, 10, 25, 44, 71].”
Reversible Architectures for Arbitrarily Deep Residual Neural Networks (2017, background): “The connection between nonlinear ODEs and deep ResNets has been established in the recent works of (E 2017; Haber and Ruthotto 2017; Haber, Ruthotto, and Holtham 2017; Lu et al. 2017; Long et al. 2017; Chang et al. 2017).”
Data Driven Governing Equations Approximation Using Deep Neural Networks (2018, methodology): “We remark that it was pointed out empirically in [9] that using multiple ResNet blocks can result in networks with smaller norm.”