Reversible Architectures for Arbitrarily Deep Residual Neural Networks

By Bo-Hao Chang, Lili Meng, Eldad Haber, Lars Ruthotto, David Begert, Elliot Holtham. 2017

298 citations recorded by the source corpus.

Recently, deep residual networks have been successfully applied in many computer vision and natural language processing tasks, pushing the state-of-the-art performance with deeper and wider architectures. In this work, we interpret deep residual networks as ordinary differential equations (ODEs), which have long been studied in mathematics and physics with rich theoretical and empirical success. From this interpretation, we develop a theoretical framework on stability and reversibility of deep neural networks, and derive three reversible neural network architectures that can go arbitrarily deep in theory. The reversibility property allows a memory-efficient implementation, which does not need to store the activations for most hidden layers. Together with the stability of our architectures, this enables training deeper networks using only modest computational resources. We provide both theoretical analyses and empirical results. Experimental results demonstrate the efficacy of our architectures against several strong baselines on CIFAR-10, CIFAR-100 and STL-10 with superior or on-par state-of-the-art performance. Furthermore, we show our architectures yield superior results when trained using fewer training data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Model Architectures

Citations in the corpus

Neural Ordinary Differential Equations (2018, neutral): “The interpretation of residual networks He et al. (2016a) as approximate ODE solvers spurred research into exploiting reversibility and approximate computation in ResNets (Chang et al., 2017; Lu et al., 2017).”
Invertible Residual Networks (2018, background): “Note, that the condition is not necessary, e.g. other approaches [9, 10, 16, 6, 17] rely on triangular structures to create analytical inverses.”
Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations (2017, methodology): “Similar observations were also made by Chang et al. (2017); they designed a reversible architecture to grant stability to the dynamic system.”
Multi-level Residual Networks from Dynamical Systems View (2017, background): “Imposing structural constraints on dynamical systems such as Hamiltonian systems to conserve the energy is explored in Haber & Ruthotto (2017); Chang et al. (2017).”
JFB: Jacobian-Free Backpropagation for Implicit Networks (2021, background): “Neural ODEs leverage known connections between deep residual models and discretizations of differential equations (Haber and Ruthotto 2017; Weinan 2017; Ruthotto and Haber 2019; Chang et al. 2018; Finlay et al. 2020; Lu et al. 2018), and replace these discretizations by black-box ODE solvers in forw”
Monge-Ampère Flow for Generative Modeling (2018, methodology): “all amount, as shown in ﬁgure 1. Therefore, the numerical integration of ODE is equivalent to the forward evaluation of a deep residual neural network He et al. (2016); E (2017); Li & Shi (2017); Chang et al. (2017); Lu et al. (2017); Rousseau & Fablet (2018); Ruthotto & Haber (2018); Ha”
Functional Gradient Boosting based on Residual Network Perception (2018, background): “Moreover, several studies (Weinan, 2017; Haber et al., 2017; Chang et al., 2017a;b; Lu et al., 2017) have pointed out that ResNet layers can be regarded as discretization steps of ordinary differential equations.”
Sparsely Aggregated Convolutional Networks (2018, background): “[2], taking inspiration from ordinary differential equations, develop a framework for analyzing stability of reversible networks [8] and demonstrate very deep reversible architectures.”
Continuous-in-Depth Neural Networks (2020, background): “Among the many remarkable developments in machine learning (ML) that have taken place within the last decade, residual neural networks (ResNets) [31, 32] and their variants [81, 36, 42, 21, 8, 86, 80] are especially prominent.”
Residual Networks as Flows of Diffeomorphisms (2019, background): “Several other reversible architectures have been recently proposed [7,10,17], showing the potential of such frameworks for the analysis of residual networks.”
Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network (2018, background): “On the other hand, considering the existing deep neurally-inspired architectures as the numerical methods for solving their respective differential equations aided by the recent advances in memory-efficient implementations has helped to successfully stabilize very large models at lower computational”
Analysis of Invariance and Robustness via Invertibility of ReLU-Networks (2018, background): “Moreover, several reversible network structures were recently proposed [Gomez et al., 2017] [Chang et al., 2018] [Jacobsen et al., 2018].”
ISNet: Shape Matters for Infrared Small Target Detection (2022, background): “[3] analyzed the similarity of different neural networks and ODE.”
Deep Neural Networks Motivated by Partial Differential Equations (2018, methodology): “to avoid storage of intermediate network states, thus achieving higher memory eciency. This is particularly important for very deep networks where memory limitation can hinder training (see [16] and [9]). Hamiltonian CNNs. Introducing an auxiliary variable Z (i.e., by partitioning the channels of th”