Intermediate Layer Optimization for Inverse Problems using Deep Generative Models

By Giannis Daras, Joseph Dean, Ajil Jalal, A. Dimakis. 2021

105 citations recorded by the source corpus.

We propose Intermediate Layer Optimization (ILO), a novel optimization algorithm for solving inverse problems with deep generative models. Instead of optimizing only over the initial latent code, we progressively change the input layer obtaining successively more expressive generators. To explore the higher dimensional spaces, our method searches for latent codes that lie within a small $l_1$ ball around the manifold induced by the previous layer. Our theoretical analysis shows that by keeping the radius of the ball relatively small, we can improve the established error bound for compressed sensing with deep generative models. We empirically show that our approach outperforms state-of-the-art methods introduced in StyleGAN-2 and PULSE for a wide range of inverse problems including inpainting, denoising, super-resolution and compressed sensing.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, background): “These properties c include (but are not limited to2) labels [61], text descriptions [60, 76], noisy or partial observations [11, 5, 47, 24], and manipulation constraints [69].”
Robust Compressed Sensing MRI with Deep Generative Priors (2021, background): “The approach in [13] has been generalized to tackle different inverse problems [45, 34, 7, 67, 56, 59, 70, 9], and different reconstruction algorithms [21, 48, 65, 27, 26, 60, 36, 37, 18].”
Denoising Diffusion Restoration Models (2022, unknown): “Multiple such methods were suggested, mainly focusing on generative adversarial networks (GANs) (Bora et al., 2017; Daras et al., 2021; Menon et al., 2020).”
Prior Image-Constrained Reconstruction using Style-Based Generative Models (2021, unknown): “(9) are not completely realistic, but have been made in recent literature, and there has been progress towards addressing them (G ´ omez et al., 2019; Daras et al., 2021).”
Labels4Free: Unsupervised Segmentation using StyleGAN (2021, background): “Examples include various image and video editing tasks, image inpainting [13, 47, 46, 41], local image editing [42, 2], low bit-rate video conferencing [40], image super resolution [28, 12], image colorization [4, 27], and extracting 3D models [30].”
Improving Diffusion Models for Inverse Problems using Manifold Constraints (2022, background): “The former ability often lacks in supervised learning-based methods [43, 50], and the latter is often not satisﬁed for some unsupervised GAN-based solutions [10, 4].”
DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior (2023, background): “Earlier works [2, 9, 36, 39] mainly concentrate on searching a latent code within a pre-trained GAN’s latent space.”