Deep Neural Networks Motivated by Partial Differential Equations

By Lars Ruthotto, Eldad Haber. 2018

570 citations recorded by the source corpus.

Partial differential equations (PDEs) are indispensable for modeling many physical phenomena and also commonly used for solving image processing tasks. In the latter area, PDE-based approaches interpret image data as discretizations of multivariate functions and the output of image processing algorithms as solutions to certain PDEs. Posing image processing problems in the infinite-dimensional setting provides powerful tools for their analysis and solution. For the last few decades, the reinterpretation of classical image processing problems through the PDE lens has been creating multiple celebrated approaches that benefit a vast area of tasks including image segmentation, denoising, registration, and reconstruction. In this paper, we establish a new PDE interpretation of a class of deep convolutional neural networks (CNN) that are commonly used to learn from speech, image, and video data. Our interpretation includes convolution residual neural networks (ResNet), which are among the most promising approaches for tasks such as image classification having improved the state-of-the-art performance in prestigious benchmark challenges. Despite their recent successes, deep ResNets still face some critical challenges associated with their design, immense computational costs and memory requirements, and lack of understanding of their reasoning. Guided by well-established PDE theory, we derive three new ResNet architectures that fall into two new classes: parabolic and hyperbolic CNNs. We demonstrate how PDE theory can provide new insights and algorithms for deep learning and demonstrate the competitiveness of three new CNN architectures using numerical experiments.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Model Architectures, Numerical Methods & Scientific Computing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Model Architectures

In Numerical Methods & Scientific Computing

Citations in the corpus

Neural Ordinary Differential Equations (2018, neutral): “…and normalizing ﬂ ows build complicated transformations by composing a sequence of transformations to a hidden state: where { 0 � � These iterative updates can be seen as an Euler discretization of a continuous transformation (Lu et al., 2017; Haber and Ruthotto, 2017; Ruthotto and Haber, 2018).”
Invertible Residual Networks (2018, methodology): “Thus, the invertible ResNet with Lip( f ) 1 can be understood as a stabilizer of an ODE for step size h = 1 without a restriction to anti-symmetric layers as in [20, 14, 8].”
Augmented Neural ODEs (2019, neutral): “The relationship between neural networks and differential equations has been studied in several recent works (Weinan, 2017; Lu et al., 2017; Haber & Ruthotto, 2017; Ruthotto & Haber, 2018; Chen et al., 2018).”
Spherical CNNs on Unstructured Grids (2019, background): “Ruthotto & Haber (2018) explore the reparameterization of convolutional kernels using parabolic and hyperbolic differential basis with regular grid images.”
Continuous-in-Depth Neural Networks (2020, background): “In recent literature, so-called Neural ODEs have attracted attention in both the ML and the physical sciences communities [11, 66, 16, 87, 85, 20].”
Integrating Physics-Based Modeling with Machine Learning: A Survey (2020, background): “In a pioneering work by Ruthotto et al [218], three variations of CNNs are proposed for solving PDEs.”
Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network (2018, background): “On one hand, leveraging the established mathematical theories from differential equations in the continuous-time domain has historically led to a better understanding of the evolution of the related difference equations, since the difference equations are obtained from the corresponding original dif”
Solving inverse problems using data-driven models (2019, background): “7) for learned reaction–diffusion equations), deriving stable architectures by drawing connections to ordinary differential equations (Haber and Ruthotto 2017) and constraining CNNs (Ruthotto and Haber 2018) by the interpretation as a partial differential”
PDEBENCH: An Extensive Benchmark for Scientific Machine Learning (2022, methodology): “Methods include Physics-informed neural networks (PINNs) [47], Neural operators (NOs) [32, 26], treating ResNet as a PDE approximant [49], custom architectures for specific problems such as TFNet for turbulent fluid flows [61], and generic image-to-image regression models such as the U-Net [48].”
JFB: Jacobian-Free Backpropagation for Implicit Networks (2021, background): “…ODEs leverage known connections be-tween deep residual models and discretizations of differential equations (Haber and Ruthotto 2017; Weinan 2017; Ruthotto and Haber 2019; Chang et al. 2018; Finlay et al. 2020; Lu et al. 2018), and replace these discretizations by black-box ODE solvers in…”
Scalable Gradients for Stochastic Differential Equations (2020, methodology): “[11] used this method to compute the gradient with respect to parameters of a neural ODE, which is a particular model among many others inspired by the theory of dynamical systems [9, 10, 25, 44, 71].”
Residual Networks as Flows of Diffeomorphisms (2019, background): “By contrast, considering higher-complexity elementary blocks in a ResNet architectures without shared weights would increase numerical instabilities and may required complementary regularization constraints across network depth [15, 25].”
ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs (2019, background): “The connection between residual networks and ODEs has been discussed in [10, 20, 23, 25, 27].”
Physics-aware Difference Graph Networks for Sparsely-Observed Dynamics (2020, background): “Ruthotto & Haber (2018) derived new CNNs: parabolic and hyperbolic CNNs based on ResNet (He et al., 2016) architecture motivated by PDE theory.”
How to train your neural ODE (2020, background): “Optimal transport map and a generic normalizing flow. between ResNets and the numerical solution of ordinary differential equations (Haber & Ruthotto, 2017; Ruthotto & Haber, 2018; Chen et al., 2018; 2019).”
Learning Differential Equations that are Easy to Solve (2020, methodology): “Solving the resulting IVP is analogous to evaluating a very deep residual network in which the number of layers corresponds to the number of function evaluations of the solver (Chang et al., 2017; Ruthotto & Haber, 2018; Chen et al., 2018).”
Integrating Scientific Knowledge with Machine Learning for Engineering and Environmental Systems (2020, background): “5) [226] [251] [301] [303] [70] [196] [228] [248] [318] [82] [102] [144] [292] [216] [195] [23] [90] [320] [154] [58] [236] [53] [70] [192] [252] [26] [61] [149] [88] [299] [216] [195] [225] [145] [198] [171] [44] [188]”