Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations

By Yiping Lu, Aoxiao Zhong, Quanzheng Li, Bin Dong. 2017

565 citations recorded by the source corpus.

In our work, we bridge deep neural network design with numerical differential equations. We show that many effective networks, such as ResNet, PolyNet, FractalNet and RevNet, can be interpreted as different numerical discretizations of differential equations. This finding brings us a brand new perspective on the design of effective deep architectures. We can take advantage of the rich knowledge in numerical analysis to guide us in designing new and potentially more effective deep networks. As an example, we propose a linear multi-step architecture (LM-architecture) which is inspired by the linear multi-step method solving ordinary differential equations. The LM-architecture is an effective structure that can be used on any ResNet-like networks. In particular, we demonstrate that LM-ResNet and LM-ResNeXt (i.e. the networks obtained by applying the LM-architecture on ResNet and ResNeXt respectively) can achieve noticeably higher accuracy than ResNet and ResNeXt on both CIFAR and ImageNet with comparable numbers of trainable parameters. In particular, on both CIFAR and ImageNet, LM-ResNet/LM-ResNeXt can significantly compress ($>50$\%) the original networks while maintaining a similar performance. This can be explained mathematically using the concept of modified equation from numerical analysis. Last but not least, we also establish a connection between stochastic control and noise injection in the training process which helps to improve generalization of the networks. Furthermore, by relating stochastic training strategy with stochastic dynamic system, we can easily apply stochastic training to the networks with the LM-architecture. As an example, we introduced stochastic depth to LM-ResNet and achieve significant improvement over the original LM-ResNet on CIFAR10.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Model Architectures

Citations in the corpus

Learning Deep Transformer Models for Machine Translation (2019, methodology): “Also, recent work shows successful applications of LMM in computer vision, but only two previous steps are used in their LMM-like system (Lu et al., 2018).”
Neural Ordinary Differential Equations (2018, neutral): “Finally, we derived an instantaneous version of the change of variables formula, and developed continuous-time normalizing ﬂ ows, which can scale to large layer sizes.”
Multistep Neural Networks for Data-driven Discovery of Nonlinear Dynamical Systems (2018, background): “, irregularly sampled data in time? How would common techniques such as batch normalization, drop out, and L1/L2 regularization enhance the robustness of the proposed algorithm and mitigate the effects of over-fitting? How could one incorporate partial knowledge of the dynamical system in cases wher”
Invertible Residual Networks (2018, methodology): “…and Euler-methods from ODEs with activations (states) x t ∈ R d , layer-index (time-step) t = 1 , . . . , T , scaling (step-size) h > 0 and residual block g θ t (dynamic of ODE f θ t ( x t ) ) has attracted growing research in the intersection of deep learning and dynamical systems [18, 14, 20, 7].”
Augmented Neural ODEs (2019, neutral): “The relationship between neural networks and differential equations has been studied in several recent works (Weinan, 2017; Lu et al., 2017; Haber & Ruthotto, 2017; Ruthotto & Haber, 2018; Chen et al., 2018).”
PDE-Net: Learning PDEs from Data (2017, background): “An interesting fact is that some popular networks in computer vision, such as ResNet(He et al., 2016a;b), have close relationship with PDEs (Chen et al., 2015; E, 2017; Haber & Ruthotto, 2017; Sonoda & Murata, 2017; Lu et al., 2017).”
Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise (2019, methodology): “A framework was proposed to link existing deep architectures with discretized numerical ODE solvers [9], and was shown to be parameter efficient.”
Disentangling Physical Dynamics From Unknown Factors for Unsupervised Video Prediction (2020, methodology): “Euler, Runge-Kutta) and residual neural networks [71, 37, 8, 78], several specific architectures were designed for predicting and identifying dynamical systems [15, 35, 47].”
ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs (2019, background): “The connection between residual networks and ODEs has been discussed in [10, 20, 23, 25, 27].”
PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid Deep Network (2018, methodology): “Inspired by the dynamic system perspective of deep neural networks [17, 18, 19, 20, 21, 22], we consider forward Euler as the temporal discretization of the evolution PDE (1), and unroll the discrete dynamics to a feed-forward network.”
JFB: Jacobian-Free Backpropagation for Implicit Networks (2021, background): “Neural ODEs leverage known connections between deep residual models and discretizations of differential equations (Haber and Ruthotto 2017; Weinan 2017; Ruthotto and Haber 2019; Chang et al. 2018; Finlay et al. 2020; Lu et al. 2018), and replace these discretizations by black-box ODE solvers in forw”
Monge-Ampère Flow for Generative Modeling (2018, background): “Of particular interests are the variational autoencoders (VAEs) Kingma & Welling (2013); Rezende et al. (2014), generative adversarial networks (GANs) Goodfellow et al.”
Functional Gradient Boosting based on Residual Network Perception (2018, background): “Moreover, several studies (Weinan, 2017; Haber et al., 2017; Chang et al., 2017a;b; Lu et al., 2017) have pointed out that ResNet layers can be regarded as discretization steps of ordinary differential equations.”
Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network (2018, background): “On one hand, leveraging the established mathematical theories from differential equations in the continuous-time domain has historically led to a better understanding of the evolution of the related difference equations, since the difference equations are obtained from the corresponding original dif”
Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View (2019, methodology): “As we can see, this is mathematically equivalent to the ResNet architecture [25, 8]: The function γf ( x l , t l ) can be considered as a neural-network block, and the second argument t l in the function indicates the set of parameters in the l -th layer.”
Continuous-in-Depth Neural Networks (2020, methodology): “Due to their popularity, there is a rich body of work that aims to explain “why” shortcut connections between layers enables the training of very deep NNs [73, 4, 29, 48, 51, 44].”
Scalable Gradients for Stochastic Differential Equations (2020, methodology): “[11] used this method to compute the gradient with respect to parameters of a neural ODE, which is a particular model among many others inspired by the theory of dynamical systems [9, 10, 25, 44, 71].”
Stochastic Normalizing Flows (2020, background): “In a general machine learning context, one might choose f such that the Euler discretization of (1) resembles layer-wise updates of a residual neural network (Lu et al., 2017; Chen et al., 2018), or one may parameterize f as a neural network itself (Grathwohl et al.”