NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations

By Marco Ciccone, Marco Gallieri, Jonathan Masci, Christian Osendorfer, Faustino J. Gomez. 2018

59 citations recorded by the source corpus.

This paper introduces "Non-Autonomous Input-Output Stable Network" (NAIS-Net), a very deep architecture where each stacked processing block is derived from a time-invariant non-autonomous dynamical system. Non-autonomy is implemented by skip connections from the block input to each of the unrolled processing stages and allows stability to be enforced so that blocks can be unrolled adaptively to a pattern-dependent processing depth. We prove that the network is globally asymptotically stable so that for every initial condition there is exactly one input-dependent equilibrium assuming tanh units, and multiple stable equilibria for ReLU units. An efficient implementation that enforces the stability under derived conditions for both fully-connected and convolutional layers is also presented. Experimental results show how NAIS-Net exhibits stability in practice, yielding a significant reduction in generalization gap compared to ResNets.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Michael W. Mahoney
2. Kurt Keutzer
3. A. Gholami
4. Jörn-Henrik Jacobsen
5. David Kristjanson Duvenaud
6. Jens Behrmann
7. N. Benjamin Erichson
8. Zi-Mei Yao

In Electrical and Electronic Engineering

1. Kurt Keutzer
2. A. Gholami
3. Michael W. Mahoney
4. Patrick Kidger
5. Terry Lyons
6. David Kristjanson Duvenaud
7. George Biros
8. A. Queiruga

In Statistical and Nonlinear Physics

1. Michael W. Mahoney
2. Jörn-Henrik Jacobsen
3. Jens Behrmann
4. N. Benjamin Erichson
5. A. Gholami
6. Patrick Kidger
7. Terry Lyons
8. David Kristjanson Duvenaud

Citations in the corpus

• Invertible Residual Networks (2018, methodology): “…dynamics: The view of deep networks as dynamics over time offers two fundamental learning approaches: 1) Direct learning of continuous ODE dynamics parametrized by neural networks as in [7, 13] and 2) Indirect learning of ODE dynamics using discretization architectures like ResNets [14, 20, 18, 8].”
• Continuous-in-Depth Neural Networks (2020, methodology): “This connection was first established by [75, 25], and it has since been used as inspiration for the design of ResNet variants and algorithms, such as invertible networks [5] and stabilization [25, 26, 13].”
• Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network (2018, background): “On the other hand, considering the existing deep neurally-inspired architectures as the numerical methods for solving their respective differential equations aided by the recent advances in memory-efficient implementations has helped to successfully stabilize very large models at lower computational”
• ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs (2019, background): “The connection between residual networks and ODEs has been discussed in [10, 20, 23, 25, 27].”
• Neural Controlled Differential Equations for Irregular Time Series (2020, background): “Indeed the connection between RNNs and dynamical systems is well-known [1, 2, 3, 4].”
• UNCERTAINTY MODELING AND INTERPRETABILITY IN CONVOLUTIONAL NEURAL NETWORKS FOR POLYP SEGMENTATION (2018, background): “Database Training Validation Test Resolution CVC-Colon [5] [1-76], [98-148], [221-273].”