Image-Adaptive GAN based Reconstruction

By Shady Abu Hussein, Tom Tirer, Raja Giryes. 2019

104 citations recorded by the source corpus.

In the recent years, there has been a significant improvement in the quality of samples produced by (deep) generative models such as variational auto-encoders and generative adversarial networks. However, the representation capabilities of these methods still do not capture the full distribution for complex classes of images, such as human faces. This deficiency has been clearly observed in previous works that use pre-trained generative models to solve imaging inverse problems. In this paper, we suggest to mitigate the limited representation capabilities of generators by making them image-adaptive and enforcing compliance of the restoration with the observations via back-projections. We empirically demonstrate the advantages of our proposed approach for image super-resolution and compressed sensing.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Generative Adversarial Networks (2021, neutral): “By providing a thorough overview of GANs, we hope to make this complex and challenging topic more accessible to researchers and practitioners in the field, and inspire new innovations and applications.”
GAN Inversion: A Survey (2021, methodology): “There are also some attempts that use a pretrained generative model as a source of image statistics [18], [139].”
Robust Compressed Sensing MRI with Deep Generative Priors (2021, background): “[40] Shady Abu Hussein, Tom Tirer, and Raja Giryes.”
A Comprehensive Survey on Test-Time Adaptation Under Distribution Shifts (2023, unknown): “Besides, the TTA paradigm is introduced to image registration [421], [422], inverse problems [429], [469], [580], [581], quality assessment and enhancement [254], [295].”
Hijack-GAN: Unintended-Use of Pretrained, Black-Box GANs (2020, background): “One way is to treat pretrained GANs as strong image priors, which can be achieved by either exploiting GANs as a component of their networks [22, 5, 6, 17, 3] or directly editing the latent code to find feasible solutions [1, 2, 11].”
Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation (2020, methodology): “use a pre-trained GAN as a source of image statistics include [32] and [33], which respectively applies to image manipulation, e.”
Invertible generative models for inverse problems: mitigating representation error and dataset bias (2019, unknown): “• For compressive sensing of in-distribution images, in-vertible neural network priors can yield higher PSNRs than GANs with low-dimensional latent dimensionality (both DCGAN and PG-GAN), Image Adaptive GANs, sparsity priors, and a Deep Decoder across a wide range of undersampling ratios.”
Prior Image-Constrained Reconstruction using Style-Based Generative Models (2021, background): “Several approaches have tried to reduce the representation error arising from the use of GANs as priors (Asim et al., 2020; Hussein et al., 2020).”
HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing (2021, background): “[28] invert images into a pre-trained GAN by first recovering a latent code which approximately reconstructs the target image and then fine-tuning the generator weights for improve image-specific details.”
Improving Diffusion Models for Inverse Problems using Manifold Constraints (2022, methodology): “Score-SDE [41] 38.68 RePAINT [32] 247.6 DDRM [25] 2.117 LaMa [43] 0.629 AOT-GAN [52] 0.082 ICT [49] 144.6 DSI [35] 36.64 IAGAN [20] 518.47 Ours 81.59 Ablation studies We perform three ablation studies: 1) As both MCG term and the projection term contain information about the measurement y , we obser”