Why are deep nets reversible: A simple theory, with implications for training

By Sanjeev Arora, Yingyu Liang, Tengyu Ma. 2015

55 citations recorded by the source corpus.

Generative models for deep learning are promising both to improve understanding of the model, and yield training methods requiring fewer labeled samples. Recent works use generative model approaches to produce the deep net's input given the value of a hidden layer several levels above. However, there is no accompanying "proof of correctness" for the generative model, showing that the feedforward deep net is the correct inference method for recovering the hidden layer given the input. Furthermore, these models are complicated. The current paper takes a more theoretical tack. It presents a very simple generative model for RELU deep nets, with the following characteristics: (i) The generative model is just the reverse of the feedforward net: if the forward transformation at a layer is $A$ then the reverse transformation is $A^T$. (This can be seen as an explanation of the old weight tying idea for denoising autoencoders.) (ii) Its correctness can be proven under a clean theoretical assumption: the edge weights in real-life deep nets behave like random numbers. Under this assumption ---which is experimentally tested on real-life nets like AlexNet--- it is formally proved that feed forward net is a correct inference method for recovering the hidden layer. The generative model suggests a simple modification for training: use the generative model to produce synthetic data with labels and include it in the training set. Experiments are shown to support this theory of random-like deep nets; and that it helps the training.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Scaling & Learning Theory, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Scaling & Learning Theory

In Signal Processing

Citations in the corpus

Variational Walkback: Learning a Transition Operator as a Stochastic Recurrent Net (2017, neutral): “Such approach towards symmetry is consistent with both empirical observations (Vincent et al., 2010) and theoretical analysis (Arora et al., 2015) of auto-encoders, for which symmetric weights are associated with minimizing reconstruction error.”
DISCO: Dynamic and Invariant Sensitive Channel Obfuscation for deep neural networks (2020, background): “[55] for the client f1, following [56], to generalize the attack surface to sensitive inputs.”
AdaNet: Adaptive Structural Learning of Artificial Neural Networks (2016, background): “The fourth takes the generative point of view (e.g. [Arora et al., 2014, 2015]), assuming that the data actually comes from a particular network and then attempting to recover it.”
Toward an Integration of Deep Learning and Neuroscience (2016, background): “This schemerequiressymmetricweights,butinanautoencodercontext,Bengionotesthatthesecanarisespontaneously(Aroraetal.,2015). network across multiple discrete time steps and then runs backpropagation on the unfolded network to assign credit to particular units at particular time steps 11 .”
Towards a Biologically Plausible Backprop (2016, background): “Another encouraging piece of evidence, also linked to autoencoders, is the theoretical result from Arora et al. (2015), showing that the symmetric solution minimizes the autoencoder reconstruction error between two successive layers of rectifying (ReLU) units.”
Equilibrium Propagation: Bridging the Gap between Energy-Based Models and Backpropagation (2016, methodology): “” It is shown by Bengio and Fischer (2015) and Bengio et al. (2017) that the early change of neural activities in the second phase corresponds to the propagation of error derivatives with respect to neural activities. Our contribution in this paper is to go beyond the early change of neural activiti”
Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning (2018, methodology): “This includes: work on large-batch learning and the generalization gap [148, 72, 64, 57, 67, 131, 66, 146, 94, 151, 152]; work on Energy Landscape approaches to NN training [70, 149, 44, 123, 30, 29, 27, 65, 47, 12, 104, 150, 50, 83, 82, 95]; work on using weight matrices or properties of weight mat”
Deep Neural Networks are Surprisingly Reversible: A Baseline for Zero-Shot Inversion (2021, background): “For example, one direction of works provides theoretical proof of the invertibility assuming the weights of DNNs are random-like [19, 29, 30], and expansion rates are larger than a certain constant [21, 31].”
A Provably Convergent Scheme for Compressive Sensing Under Random Generative Priors (2018, background): “Moreover, recent papers [1,7,19,26] assume that the weights are initialized during training by Gaussian distributions and establish that with sufficient overparameterization, the weights are only perturbed slightly during the training process.”
Analysis of Invariance and Robustness via Invertibility of ReLU-Networks (2018, background): “A notable other line of work assumes random weights in order to derive guarantees for invertibility, see [Gilbert et al., 2017] or [Arora et al., 2015], whereas we aim to theoretically derive means to uncover mechanisms of rectiﬁer networks without assumptions on the weights.”
Reversible Architectures for Arbitrarily Deep Residual Neural Networks (2017, background): “This suggests that some deep neural networks are reversible: the generative model is just the reverse of the feed-forward net (Arora, Liang, and Ma 2016).”
Convolutional Neural Networks (2017, unknown): “Following the unprecedented success of deep networks, there has been some theoretical work (e.g., Arora et al. (2014; 2015); Paul and Venkatasubramanian (2014)) that suggest several mathematical models for different deep learning architectures.”
Towards Understanding the Invertibility of Convolutional Neural Networks (2017, background): “Following the unprecedented success of deep networks, there have been some theoretical works [Arora et al. , 2014; Arora et al. , 2015; Paul and Venkatasubramanian, 2014] that suggest several mathematical models for different deep learning architectures.”
Global Guarantees for Enforcing Deep Generative Priors by Empirical Risk (2017, methodology): “The results of [59] are in a setting where the neural net is not generative, and their procedure is at only approximate, and, since it requires observation of the last layer, it is not readily extendable to the compressive linear observation setting.”