Global Guarantees for Enforcing Deep Generative Priors by Empirical Risk

By Paul Hand, Vladislav Voroninski. 2017

144 citations recorded by the source corpus.

We examine the theoretical properties of enforcing priors provided by generative deep neural networks via empirical risk minimization. In particular we consider two models, one in which the task is to invert a generative neural network given access to its last layer and another in which the task is to invert a generative neural network given only compressive linear observations of its last layer. We establish that in both cases, in suitable regimes of network layer sizes and a randomness assumption on the network weights, that the non-convex objective function given by empirical risk minimization does not have any spurious stationary points. That is, we establish that with high probability, at any point away from small neighborhoods around two scalar multiples of the desired solution, there is a descent direction. Hence, there are no local minima, saddle points, or other stationary points outside these neighborhoods. These results constitute the first theoretical guarantees which establish the favorable global geometry of these non-convex optimization problems, and they bridge the gap between the empirical success of enforcing deep generative priors and a rigorous understanding of non-linear inverse problems.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

In Statistical Learning Theory

Citations in the corpus

Deep Decoder: Concise Image Representations from Untrained Non-convolutional Networks (2018, methodology): “…a generative image model (by training an autoencoder or GAN [HS06; Goo+14]) and then using the resulting model to solve inverse problems such as compressed sensing [Bor+17; HV18], denoising [Hec+18], phase retrieval [Han+18; SA18], and blind deconvolution [Asi+18], by minimizing an associated loss.”
Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overview (2018, unknown): “Furthermore, we have also left out several important nonconvex problems and methods in order to stay focused, including but not limited to (1) blind calibration and finding sparse vectors in a subspace [97, 220, 221]; (2) tensor completion, decomposition and mixture models [156, 222–225]; (3) parame”
Deep Learning Techniques for Inverse Problems in Imaging (2020, background): “mic factor, [136] proved that the objective (4) has only two local minima and can be optimized by gradient descent.”
Intermediate Layer Optimization for Inverse Problems using Deep Generative Models (2021, background): “On the theory side, a central open problem would be to establish global convergence of ILO, possibly following the ideas of (Hand & Voroninski, 2018; Hand et al., 2018) or surﬁng (Song et al., 2019).”
Robust Compressed Sensing MRI with Deep Generative Priors (2021, background): “[35] Paul Hand and Vladislav Voroninski.”
Your Local GAN: Designing Two Dimensional Local Attention Mechanisms for Generative Models (2019, background): “Very recent research [13, 23] demonstrated that for fully connected generators with random weights and sufficient layer expansion, gradient descent will provably converge to the correct optimal inversion.”
When and How Can Deep Generative Models be Inverted? (2020, background): “Even though this inversion is generally non-convex, the works in [13, 12] show that if the weights are random then, under additional assumptions, no spurious stationary points exist, and thus gradient descent converges to the optimal solution.”
Neural Proximal Gradient Descent for Compressive Imaging (2018, background): “Note also that for a signal that has a low-dimensional code under a deep pre-trained generative model, [28, 29] establishes reconstruction guarantees.”
Compressive sensing with un-trained neural networks: Gradient descent finds the smoothest approximation (2020, background): “…denoising [Bur+12; Zha+17], or are based on learning a generative image model (by training an autoencoder or GAN [HS06; Goo+14]) and then using the resulting image models to regularize problems such as compressed sensing [Bor+17; HV18; Hua+18], denoising [Hec+20], or phase retrieval [Han+18; SA18].”
Inverting Deep Generative models, One layer at a time (2019, methodology): “Recovery with Various Input Neurons: According to the theoretical result, one advantage of our proposals is the much smaller expansion requirement than gradient descent [12] (constant vs log k factors).”
Invertibility of Convolutional Generative Networks from Partial Measurements (2018, methodology): “4, which deviates from the proof machinery in [8] and hence is a major technical contribution.”
Image Processing Using Multi-Code GAN Prior (2019, unknown): “Some work theoretically explored the prior provided by deep generative models [32, 18], but the results using GAN prior to real image processing are still unsatisfying.”