The Reversible Residual Network: Backpropagation Without Storing Activations

By Aidan N. Gomez, Mengye Ren, Raquel Urtasun, Roger Baker Grosse. 2017

654 citations recorded by the source corpus.

Deep residual networks (ResNets) have significantly pushed forward the state-of-the-art on image classification, increasing in performance as networks grow both deeper and wider. However, memory consumption becomes a bottleneck, as one needs to store the activations in order to calculate gradients using backpropagation. We present the Reversible Residual Network (RevNet), a variant of ResNets where each layer's activations can be reconstructed exactly from the next layer's. Therefore, the activations for most layers need not be stored in memory during backpropagation. We demonstrate the effectiveness of RevNets on CIFAR-10, CIFAR-100, and ImageNet, establishing nearly identical classification accuracy to equally-sized ResNets, even though the activation storage requirements are independent of depth.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Momentum Contrast for Unsupervised Visual Representation Learning (2019, neutral): “net [23], RX50 is ResNeXt-50-32×8d [62].”
Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, background): “Instead, memory storage/movement is the limiting factor in image classification (Gomez et al., 2017), semantic segmentation (Chen Figure 8.”
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019, background): “Gomez et al. (2017) propose a way to reconstruct each layer’s activations from the next layer so that they do not need to store the intermediate activations.”
Reformer: The Efficient Transformer (2020, methodology): “…2020 Published as a conference paper at ICLR 2020 We introduce the Reformer model which solves these problems using the following techniques: • Reversible layers, ﬁrst introduced in Gomez et al. (2017), enable storing only a single copy of activations in the whole model, so the N factor disappears.”
Glow: Generative Flow with Invertible 1x1 Convolutions (2018, methodology): “Computing gradients in reversible neural networks requires an amount of memory that is constant instead of linear in their depth, as explained in the RevNet paper (Gomez et al., 2017).”
Neural Ordinary Differential Equations (2018, positive): “Constant memory backprop through reversibility Recent work developed reversible versions of residual networks (Gomez et al., 2017; Haber and Ruthotto, 2017; Chang et al., 2017), which gives the same constant memory advantage as our approach.”
Large Scale Adversarial Representation Learning (2019, background): “In particular, we expand the capacity of the hidden layers by a factor of 2 or 4, as well as swap the residual block structure to a reversible variant called RevNet [10] with the same number of layers and capacity as the corresponding ResNets.”
Revisiting Self-Supervised Visual Representation Learning (2019, methodology): “RevNet slightly modifies the design of the residual unit such that it becomes analytically invertible [12].”
8-bit Optimizers via Block-wise Quantization (2021, background): “They include gradient checkpointing (Chen et al., 2016), reversible residual connections (Gomez et al., 2017), and offloading (Pudipeddi et al., 2020; Rajbhandari et al., 2021).”
Tensor2Tensor for Neural Machine Translation (2018, methodology): “• tf.contrib.layers.rev block, implementing a memory-efficient block of reversible layers as presented in Gomez et al. (2017), was first implemented and exercised in Tensor2Tensor.”
Improving GANs Using Optimal Transport (2018, methodology): “We also investigated forcing v η to be one-to-one by parameterizing it using a RevNet Gomez et al. (2017), thereby ensuring c cannot degenerate, but this proved unnecessary if the generator is updated often enough.”
Graph Normalizing Flows (2019, methodology): “In the supervised case, we leverage a similar mechanism to [7] to obtain signiﬁcant memory savings in a model we call reversible graph neural networks, or GRevNets.”
Excessive Invariance Causes Adversarial Vulnerability (2018, methodology): “We also leverage recent advances in reversible (Gomez et al., 2017) and bijective networks (Jacobsen et al., 2018; Ardizzone et al., 2018; Kingma & Dhariwal, 2018) for our analysis.”
Invertible Residual Networks (2018, background): “The ability to invert deep networks alleviates the need to store intermediate activations during backpropagation [11, 6] and offers a promising approach in Inverse Problems [3].”
Representing smooth functions as compositions of near-identity functions with implications for deep network optimization (2018, background): “The practical strength, and therefore relevance, of invertible feature maps in the non-linear case is supported by success the reversible networks (Maclaurin et al., 2015; Gomez et al., 2017; Dinh et al., 2017).”
i-RevNet: Deep Invertible Networks (2018, methodology): “For the same reasons as in Gomez et al. (2017), our scheme also allows avoiding storing any intermediate activations at training time, making memory consumption for very deep i -RevNets not an issue in practice.”
Adversarial Logit Pairing (2018, methodology): “For SVHN, we used the RevNet-9 model (Gomez et al., 2017).”
In-place Activated BatchNorm for Memory-Optimized Training of DNNs (2017, background): “In [7] the authors modify ResNet in a way to contain reversible residual blocks, i.”
ETC: Encoding Long and Structured Inputs in Transformers (2020, methodology): “…we would like to investigate complementary attention mechanisms like those of Reformer (Kitaev et al., 2020) or Routing Trans-former (Roy et al., 2020), push scalability with ideas like those from RevNet (Gomez et al., 2017), and study the performance of ETC in datasets with even richer structure.”
Neural Spline Flows (2019, background): “More generally, invertibility can be useful for training very large networks, since activations can be recomputed on-the-fly for backpropagation, meaning gradient computation requires memory which is constant instead of linear in the depth of the network [13, 36].”