Stable architectures for deep neural networks

By Eldad Haber, Lars Ruthotto. 2017

856 citations recorded by the source corpus.

Deep neural networks have become invaluable tools for supervised machine learning, e.g. classification of text or images. While often offering superior results over traditional techniques and successfully expressing complicated patterns in data, deep architectures are known to be challenging to design and train such that they generalize well to new data. Critical issues with deep architectures are numerical instabilities in derivative-based learning algorithms commonly called exploding or vanishing gradients. In this paper, we propose new forward propagation techniques inspired by systems of ordinary differential equations (ODE) that overcome this challenge and lead to well-posed learning problems for arbitrarily deep networks. The backbone of our approach is our interpretation of deep learning as a parameter estimation problem of nonlinear dynamical systems. Given this formulation, we analyze stability and well-posedness of deep learning and use this new understanding to develop new network architectures. We relate the exploding and vanishing gradient phenomenon to the stability of the discrete ODE and present several strategies for stabilizing deep learning for very deep networks. While our new architectures restrict the solution space, several numerical experiments show their competitiveness with state-of-the-art networks.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Model Architectures

In Optimization & Training

Citations in the corpus

Neural Ordinary Differential Equations (2018, neutral): “…and normalizing ﬂ ows build complicated transformations by composing a sequence of transformations to a hidden state: where { 0 � � These iterative updates can be seen as an Euler discretization of a continuous transformation (Lu et al., 2017; Haber and Ruthotto, 2017; Ruthotto and Haber, 2018).”
Invertible Residual Networks (2018, methodology): “Thus, the invertible ResNet with Lip( f ) 1 can be understood as a stabilizer of an ODE for step size h = 1 without a restriction to anti-symmetric layers as in [20, 14, 8].”
Deep Equilibrium Models (2019, methodology): “Moreover, while ODE-based models [24, 11] were veriﬁed on numerical experiments and MNIST classiﬁcation, computation and numerical stability issues challenge their application to large-scale problems.”
Augmented Neural ODEs (2019, neutral): “The relationship between neural networks and differential equations has been studied in several recent works (Weinan, 2017; Lu et al., 2017; Haber & Ruthotto, 2017; Ruthotto & Haber, 2018; Chen et al., 2018).”
PDE-Net: Learning PDEs from Data (2017, background): “An interesting fact is that some popular networks in computer vision, such as ResNet(He et al., 2016a;b), have close relationship with PDEs (Chen et al., 2015; E, 2017; Haber & Ruthotto, 2017; Sonoda & Murata, 2017; Lu et al., 2017).”
Physical Symmetries Embedded in Neural Networks (2019, background): “Furthermore, it has been shown that NN structures with embedded constraints inspired by symplectic numerical integrators can remedy issues like the exploding and vanishing gradients in recurrent NNs [20].”
JFB: Jacobian-Free Backpropagation for Implicit Networks (2021, background): “Neural ODEs leverage known connections between deep residual models and discretizations of differential equations (Haber and Ruthotto 2017; Weinan 2017; Ruthotto and Haber 2019; Chang et al. 2018; Finlay et al. 2020; Lu et al. 2018), and replace these discretizations by black-box ODE solvers in forw”
Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network (2018, background): “On one hand, leveraging the established mathematical theories from differential equations in the continuous-time domain has historically led to a better understanding of the evolution of the related difference equations, since the difference equations are obtained from the corresponding original dif”
Deep Neural Networks Motivated by Partial Differential Equations (2018, methodology): “are most closely related to anisotropic ltering [33, 40, 9]. Parabolic CNNs allow reducing the computational costs and increasing the robustness of learning using multiscale and multilevel strategies [17, 18]. Second, hyperbolic CNNs that are inspired by Hamiltonian systems are presented and nally, ”
Multi-level Residual Networks from Dynamical Systems View (2017, background): “Imposing structural constraints on dynamical systems such as Hamiltonian systems to conserve the energy is explored in Haber & Ruthotto (2017); Chang et al. (2017).”
NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations (2018, methodology): “Moreover, stability problems in both the forward as well the backward path of VDNNs have been addressed by relying on well-known analytical approaches for continuous-time ODEs [17, 5].”
PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid Deep Network (2018, methodology): “An interesting fact is that some popular networks in computer vision, such as ResNet[15, 16], have close relationship with ODEs/PDEs and can be naturally merged with traditional computational mathematics in various tasks [17, 18, 19, 20, 21, 22, 23, 24, 25, 26].”
Continuous-in-Depth Neural Networks (2020, methodology): “Recent work has established an interesting connection between ResNets and dynamical systems [25, 9, 83, 66, 85, 82].”
GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series (2019, background): “An interesting line of research has focused on deriving better deep architectures motivated by the stability of the corresponding patial differential equations (PDE) (Haber & Ruthotto, 2017; Chang et al., 2019).”
ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs (2019, background): “In particular, Hamiltonian ODEs and the corresponding discrete systems [20] allow for stable reversibility in both continuous and discrete settings.”