Invertible Residual Networks

By Jens Behrmann, David Kristjanson Duvenaud, Jörn-Henrik Jacobsen. 2018

728 citations recorded by the source corpus.

We show that standard ResNet architectures can be made invertible, allowing the same model to be used for classification, density estimation, and generation. Typically, enforcing invertibility requires partitioning dimensions or restricting network architectures. In contrast, our approach only requires adding a simple normalization step during training, already available in standard frameworks. Invertible ResNets define a generative model which can be trained by maximum likelihood on unlabeled data. To compute likelihoods, we introduce a tractable approximation to the Jacobian log-determinant of a residual block. Our empirical evaluation shows that invertible ResNets perform competitively with both state-of-the-art image classifiers and flow-based generative models, something that has not been previously achieved with a single architecture.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Stochastic Normalizing Flows (2020, background): “, 2019), and (ii) residual flows (Chen et al., 2018; Zhang et al., 2018; Grathwohl et al., 2018; Behrmann et al., 2018; Chen et al., 2019) – see (Papamakarios et al.”
Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One (2019, background): “Recent work (Behrmann et al., 2018; Chen et al., 2019) attempts to improve the discriminative performance of generative models by leveraging invertible architectures, but these methods still underperform their purely discriminative counterparts jointly trained as generative models.”
Flow Contrastive Estimation of Energy-Based Models (2019, methodology): “Recently, ﬂow-based models (henceforth simply called ﬂow models ) have gained popularity as a type of deep generative model [7, 8, 32, 17, 3, 37, 68, 10] and for use in variational inference [34, 59, 33].”
Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model (2019, methodology): “We also show that the learned short-run MCMC can be used as a generative model, such as a generator model [13, 24] or the flow model [8, 9, 25, 4, 14], with the Langevin dynamics serving as a noiseinjected residual network, with the initial image serving as the latent variables, and with the initial”
Score-Based Generative Modeling through Stochastic Differential Equations (2020, neutral): “…et al., 2016) 3.49 - iResNet (Behrmann et al., 2019) 3.45 - Glow (Kingma & Dhariwal, 2018) 3.35 - MintNet (Song et al., 2019b) 3.32 - Residual Flow (Chen et al., 2019) 3.28 46.37 FFJORD (Grathwohl et al., 2018) 3.40 - Flow++ (Ho et al., 2019) 3.29 - DDPM ( L ) (Ho et al., 2020) ď 3.70 * 13.51 DDPM…”
Revisiting Self-Supervised Visual Representation Learning (2019, background): “We hypothesize that this is a result of skip-connections making residual units invertible under certain circumstances [1], hence facilitating the preservation of information across the depth even when it is irrelevant for the pretext task.”
Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness (2020, methodology): “, gradient penalty or norm-preserving activation [1, 2, 12, 28]), and we chose spectral normalization in this work due to its simplicity and its minimal impact on a DNN’s architecture and the optimization dynamics [3, 5, 62].”
Exploiting Excessive Invariance caused by Norm-Bounded Adversarial Robustness (2019, background): “One promising architecture class to control invariancebased robustness are invertible networks (Dinh et al., 2014) because, by construction, they cannot build up any invariance until the final layer (Jacobsen et al., 2018; Behrmann et al., 2019).”
Equivariant Flows: exact likelihood generative learning for symmetric densities (2020, background): “…al., 2018), which are a subclass of autoregressive flows (Germain et al., 2015; Papamakarios et al., 2017; Huang et al., 2018; De Cao et al., 2019; Durkan et al., 2019), and 2) residual flows (Chen et al., 2018; Zhang et al., 2018; Grathwohl et al., 2018; Behrmann et al., 2018; Chen et al., 2019).”
Equivariant Flows: sampling configurations for multi-body systems with symmetric energies (2019, methodology): “2 above), BGs require an exact likelihood generative model. This can be achieved by transforming a simple prior density q Z(z), e.g. a multivariate Normal distribution, via an invertible function f [4, 5, 14, 13, 10, 2, 7, 6]. Sampling from q X;(x) is achieved by sampling z ˘q Z(z) and transforming”
Continual Learning via Local Module Composition (2021, methodology): “[10] Jens Behrmann, Will Grathwohl, Ricky TQ Chen, David Duvenaud, and Jörn-Henrik Jacobsen.”
Memory-Efficient Incremental Learning Through Feature Adaptation (2020, background): “Furthermore, this representation is more privacy preserving if we store the features separately from the network as inverting images from feature descriptors remains a challenging problem [2].”
Excessive Invariance Causes Adversarial Vulnerability (2018, background): “The role of invariance has been studied extensively in the context of deep neural networks (Bruna & Mallat, 2013; Mallat, 2016; Achille & Soatto, 2018; Behrmann et al., 2018; Cadena et al., 2018).”
Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift (2019, methodology): “Methods which model the joint distribution p ( y, x ) , e.g. deep hybrid models (Kingma et al., 2014; Alemi et al., 2018; Nalisnick et al., 2019; Behrmann et al., 2018).”
Normalizing Flows for Probabilistic Modeling and Inference (2019, methodology): “, 2019) and residual (Behrmann et al., 2019; Chen et al., 2019) flows have been demonstrated to produce sharp, high-dimensional images as well.”
Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty (2019, methodology): “Other architectural changes, such as using a Wide RevNet [Behrmann et al., 2018] instead of a Wide ResNet, can increase the AUROC from 65.3% to 77.5%.”
Residual Flows for Invertible Generative Modeling (2019, methodology): “Behrmann et al. (2019) noted that this transformation is invertible by the Banach fixed point theorem if g is contractive, i.e. with Lipschitz constant strictly less than unity, which was enforced using spectral normalization (Miyato et al., 2018; Gouk et al., 2018).”
VFlow: More Expressive Generative Flows with Variational Data Augmentation (2020, methodology): “, 2017) can not alter all the dimensions at once, while an invertible residual block (Behrmann et al., 2019) has a bounded Lipschitz constant; 3.”
Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models (2020, methodology): “If the nonlinear block of a residual mapping is no more than 1-Lipschitz, the overall mapping is invertible, and its Jacobian can be estimated using power series expansion, the Hutchinsons trace estimator (Behrmann et al., 2018) and the Russian roulette estimator (Chen et al., 2019).”