Signal recovery from Pooling Representations

By Joan Bruna, Arthur Szlam, Yann LeCun. 2013

103 citations recorded by the source corpus.

In this work we compute lower Lipschitz bounds of lp pooling operators for p = 1, 2, ∞ as well as lp pooling operators preceded by half-rectification layers. These give sufficient conditions for the design of invertible neural network layers. Numerical experiments on MNIST and image patches confirm that pooling layers can be inverted with phase recovery algorithms. Moreover, the regularity of the inverse pooling, controlled by the lower Lipschitz constant, is empirically verified with a nearest neighbor regression.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

Citations in the corpus

Recent advances in convolutional neural networks (2015, background): “The ReLU activation function is deﬁned as: where z i,j,k is the input of the activation function at location ( i, j ) on the k -th channel.”
Unsupervised Feature Learning from Temporal Data (2015, background): “Although it has recently been shown that it is possible to recover the input when We is sufficiently redundant, reconstructing from these coefficients corresponds to solving a phase recovery problem [3] which is not possible with our linear inverse map Wd.”
i-RevNet: Deep Invertible Networks (2018, methodology): “Bruna et al. (2013) derive conditions under which pooling representations are, but our method directly overcomes this issue.”
Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units (2016, methodology): “Bruna et al. (2013) theoretically investigate general conditions under which the max-pooling layer followed by ReLU is injective and measure stability of the inverting process by computing the Lipschitz lower bound.”
Unsupervised Learning of Spatiotemporally Coherent Metrics (2014, background): “Although it has recently been shown that it is possible to recover the input when We is sufficiently redundant, reconstructing from these coefficients corresponds to solving a phase recovery problem [4] which is not possible with a simple inverse mapping, such as a linear map Wd.”
A Study on Overfitting in Deep Reinforcement Learning (2018, background): “They are shown to be able to recover the original inputs from hidden representations (Bruna et al., 2014; Gilbert et al., 2017) and memorize random labels for images (Zhang et al., 2017).”
Super-Resolution with Deep Convolutional Sufficient Statistics (2015, background): “Recent work on signal recovery from these non-linear representations (Bruna et al. (2014)) suggests that the effective dimensionality of this set of admissible phases ranges from low (when the redundancy of the network is high) to nearly the ambient dimension (when the redundancy is low), thus…”
Striving for Simplicity: The All Convolutional Net (2014, background): “Since dimensionality reduction is performed via strided convolution rather than max-pooling in our architecture it also naturally lends itself to studying questions about the invertibility of neural networks (Estrach et al., 2014).”
Towards Understanding the Invertibility of Convolutional Neural Networks (2017, background): “Bruna et al. [2014] studied signal discovery from generalized pooling operators us-ing image patches on non-convolutional small scale networks and datasets.”
Deep Neural Networks are Surprisingly Reversible: A Baseline for Zero-Shot Inversion (2021, background): “Since it is impossible to invert a max pooling without additional information [98], we invert the sequence of the ﬁrst four layers as a whole block, which we refer to as the initial block.”
Optimizing the Latent Space of Generative Networks (2017, background): “From a theoretical perspective, Bruna et al. [2013] explore the theoretical conditions for a network to be invertible.”
Convolutional Neural Networks (2017, unknown): “Bruna et al. (2014) studied signal discovery from generalized pooling operators using image patches on non-convolutional small scale networks and datasets.”
Analysis of Invariance and Robustness via Invertibility of ReLU-Networks (2018, methodology): “For the purpose of studying all possible local behaviors, we deﬁne admissible index sets I l following, with slight modiﬁcations, [Bruna et al., 2014]: Of special interest for such an analysis is the range of possible effects by the application of the rectiﬁer.”