i-RevNet: Deep Invertible Networks

By Jörn-Henrik Jacobsen, Arnold M. W. Smeulders, Edouard Oyallon. 2018

360 citations recorded by the source corpus.

It is widely believed that the success of deep convolutional networks is based on progressively discarding uninformative variability about the input with respect to the problem at hand. This is supported empirically by the difficulty of recovering images from their hidden representations, in most commonly used network architectures. In this paper we show via a one-to-one mapping that this loss of information is not a necessary condition to learn representations that generalize well on complicated problems, such as ImageNet. Via a cascade of homeomorphic layers, we build the i-RevNet, a network that can be fully inverted up to the final projection onto the classes, i.e. no information is discarded. Building an invertible architecture is difficult, for one, because the local inversion is ill-conditioned, we overcome this by providing an explicit inverse. An analysis of i-RevNets learned representations suggests an alternative explanation for the success of deep networks by a progressive contraction and linear separation with depth. To shed light on the nature of the model learned by the i-RevNet we reconstruct linear interpolations between natural image representations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

On the Relationship between Self-Attention and Convolutional Layers (2019, methodology): “In all experiments, we use a 2 × 2 invertible down-sampling (Jacobsen et al., 2018) on the input to reduce the size of the image.”
Revisiting Self-Supervised Visual Representation Learning (2019, methodology): “We note that the residual unit used in [12] is equivalent to double application of the residual unit from [20] or [6].”
WT5?! Training Text-to-Text Models to Explain their Predictions (2020, background): “C L ] 3 0 A pr 2 02 0 Smilkov et al., 2017) or by designing architectures that are easier to analyze (Foerster et al., 2017; Jacobsen et al., 2018; Bahdanau et al., 2014; Raffel et al., 2017).”
Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness (2020, background): “…the fields of dimensionality reduction (Blum, 2006; Perrault-Joncas, 2017), generative modeling (Lawrence and Candela, 2006; Dinh et al., 2014; 2016; Jacobsen et al., 2018b), and adversarial robustness (Jacobsen et al., 2018a; Ruan et al., 2018; Sokolic et al., 2017; Tsuzuku et al., 2018; Weng et…”
Exploiting Excessive Invariance caused by Norm-Bounded Adversarial Robustness (2019, background): “One promising architecture class to control invariancebased robustness are invertible networks (Dinh et al., 2014) because, by construction, they cannot build up any invariance until the final layer (Jacobsen et al., 2018; Behrmann et al., 2019).”
The Incomplete Rosetta Stone problem: Identifiability results for Multi-view Nonlinear ICA (2019, background): “, the use of normalizing flows [33, 8] or deep invertible networks [24].”
Simple and Scalable Epistemic Uncertainty Estimation Using a Single Deep Deterministic Neural Network (2020, background): “Without dimensionality reduction, such as max pooling, the memory usage of these type of networks is unreasonably high (Jacobsen et al., 2018) .”
Invertible Residual Networks (2018, methodology): “Note, that the condition is not necessary, e.g. other approaches [9, 10, 16, 6, 17] rely on triangular structures to create analytical inverses.”
Similarity of Neural Network Representations Revisited (2019, background): “Moreover, in fully invertible neural networks (Dinh et al., 2017; Jacobsen et al., 2018), the mutual information between any two layers is equal to the entropy of the input.”
Normalizing Flows for Probabilistic Modeling and Inference (2019, methodology): “Jacobsen et al. (2018) showed that an invertible ResNet could be trained to nearly the same ImageNet accuracy as a non-invertible ResNet.”
Disentanglement by Nonlinear ICA with General Incompressible-flow Networks (GIN) (2020, methodology): “Downsampling increases the number of channels by a factor of 4 and decreases the image width and height by a factor of 2, done in a checkerboard-like manner, as described in Jacobsen et al. (2018).”
VFlow: More Expressive Generative Flows with Variational Data Augmentation (2020, neutral): “The bottleneck problem is studied for discriminative invert-ible models including neural ODEs (Dupont et al., 2019) and i-RevNets (Jacobsen et al., 2018), where zeros are padded to the input data to increase the number of dimensions.”
Inverting Gradients - How easy is it to break privacy in federated learning? (2020, background): “general image composition, dominating colors - but, depending on the given layer it only reconstructs similar images - if the neural network is not explicitly chosen to be (mostly) invertible [11].”
Residual Networks as Flows of Diffeomorphisms (2019, background): “Similarly to [17], the experimental re-sults show that progressive dimension changes of the embedding space are not required, contrary to popular belief that the performance of deep neural networks are based on progressive removal of uninformative variability.”
Stochastic Image-to-Video Synthesis using cINNs (2021, background): “Invertible neural networks (INNs) are bjiective functions which makes them attractive for a variety of tasks, such as analyzing inverse problems [1], interpreting neural networks [23], and representation learning [34].”
CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion (2022, background): “INNs are also applied to classiﬁcation tasks to save memory and improve the features extraction ability of the backbone [5,17,20].”
Greedy Layerwise Learning Can Scale to ImageNet (2018, methodology): “, 2015) while increasing linear separability (Zeiler & Fergus, 2014; Oyallon, 2017; Jacobsen et al., 2018) of the data.”