Continuous-in-Depth Neural Networks

By A. Queiruga, N. Benjamin Erichson, Dane Taylor, Michael W. Mahoney. 2020

56 citations recorded by the source corpus.

Recent work has attempted to interpret residual networks (ResNets) as one step of a forward Euler discretization of an ordinary differential equation, focusing mainly on syntactic algebraic similarities between the two systems. Discrete dynamical integrators of continuous dynamical systems, however, have a much richer structure. We first show that ResNets fail to be meaningful dynamical integrators in this richer sense. We then demonstrate that neural network models can learn to represent continuous dynamical systems, with this richer structure and properties, by embedding them into higher-order numerical integration schemes, such as the Runge Kutta schemes. Based on these insights, we introduce ContinuousNet as a continuous-in-depth generalization of ResNet architectures. ContinuousNets exhibit an invariance to the particular computational graph manifestation. That is, the continuous-in-depth model can be evaluated with different discrete time step sizes, which changes the number of layers, and different numerical integration schemes, which changes the graph connectivity. We show that this can be used to develop an incremental-in-depth training scheme that improves model quality, while significantly decreasing training time. We also show that, once trained, the number of units in the computational graph can even be decreased, for faster inference with little-to-no accuracy drop.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Denoising Diffusion Implicit Models (2020, methodology): “It would be interesting to see if methods that decrease the discretization error in ODEs, including multi-step methods such as Adams-Bashforth (Butcher & Goodwin, 2008), could be helpful for further improving sample quality in fewer steps (Queiruga et al., 2020).”
GRAND: Graph Neural Diffusion (2021, methodology): “In future work, we intend to overcome these limitation by introducing a θ = θ ( t ) as described in (Queiruga et al., 2020; Zhang et al., 2019).”
Long Expressive Memory for Sequence Modeling (2021, unknown): “…architectures has also received considerable attention in recent years with examples such as continuous-time neural ODEs (Chen et al., 2018; Queiruga et al., 2020; 2021) and their recurrent extensions ODE-RNNs (Rubanova et al., 2019), as well as RNNs based on discretizations of ODEs (Chang…”