Towards Understanding the Invertibility of Convolutional Neural Networks

By Anna C. Gilbert, Yi Zhang, Kibok Lee, Yuting Zhang, Honglak Lee. 2017

67 citations recorded by the source corpus.

Several recent works have empirically observed that Convolutional Neural Nets (CNNs) are (approximately) invertible. To understand this approximate invertibility phenomenon and how to leverage it more effectively, we focus on a theoretical explanation and develop a mathematical model of sparse signal recovery that is consistent with CNNs with random weights. We give an exact connection to a particular model of model-based compressive sensing (and its recovery algorithms) and random-weight CNNs. We show empirically that several learned networks are consistent with our mathematical analysis and then demonstrate that with such a simple theoretical framework, we can obtain reasonable re- construction results on real images. We also discuss gaps between our model assumptions and the CNN trained for classification in practical scenarios.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

GAN Inversion: A Survey (2021, methodology): “Recent methods aim to address this issue from different theoretical perspectives, such as compressed sensing [82], [83], [84] and sparse representation [85].”
A Study on Overfitting in Deep Reinforcement Learning (2018, background): “They are shown to be able to recover the original inputs from hidden representations (Bruna et al., 2014;Gilbert et al., 2017) andmemorize random labels for images (Zhang et al.”
Neural Network Inversion in Adversarial Setting via Background Knowledge Alignment (2019, methodology): “Inverting a neural network is one important method to understand the representations learned by neural networks [19, 23, 29, 54, 66, 77].”
Adversarial Neural Network Inversion via Auxiliary Knowledge Alignment (2019, background): “ves information about the input features in each of their intermediate representations at each layer. They replace the non-invertible components of the RevNet [72] by invertible ones. Gillbert et al. [73] focus on a theoretical explanation of the invertibility of CNN and propose a mathematical model”
Compressed Sensing using Generative Models (2017, methodology): “We also take note of the related work of [18] that connects model-based compressed sensing with the invertibility of Convolutional Neural Networks.”
Analysis of Invariance and Robustness via Invertibility of ReLU-Networks (2018, background): “A notable other line of work assumes random weights in order to derive guarantees for invertibility, see [Gilbert et al., 2017] or [Arora et al., 2015], whereas we aim to theoretically derive means to uncover mechanisms of rectiﬁer networks without assumptions on the weights.”
Deep Neural Networks are Surprisingly Reversible: A Baseline for Zero-Shot Inversion (2021, background): “For example, one direction of works provides theoretical proof of the invertibility assuming the weights of DNNs are random-like [19, 29, 30], and expansion rates are larger than a certain constant [21, 31].”
Reversible Architectures for Arbitrarily Deep Residual Neural Networks (2017, background): “(Gilbert et al. 2017) provide a theoretical connection between a model-based compressive sensing and CNNs.”