High-Fidelity GAN Inversion for Image Attribute Editing

By Tengfei Wang, Yong Zhang, Yanbo Fan, Jue Wang, Qifeng Chen. 2021

305 citations recorded by the source corpus.

We present a novel highfidelity generative adversarial network (GAN) inversion framework that enables attribute editing with image-specific details well-preserved (e.g., background, appearance, and illumination). We first analyze the challenges of highfidelity GAN inversion from the perspective of lossy data compression. With a low bitrate latent code, previous works have difficulties in preserving highfidelity details in reconstructed and edited images. Increasing the size of a latent code can improve the accuracy of GAN inversion but at the cost of inferior editability. To improve image fidelity without compromising editability, we propose a distortion consultation approach that employs a distortion map as a reference for highfidelity reconstruction. In the distortion consultation inversion (DCI), the distortion map is first projected to a high-rate latent map, which then complements the basic low-rate latent code with more details via consultation fusion. To achieve high-fidelity editing, we propose an adaptive distortion alignment (ADA) module with a self-supervised training scheme, which bridges the gap between the edited and inversion images. Extensive experiments in the face and car domains show a clear improvement in both inversion and editing quality. The project page is https://tengfei-wang.github.io/HFGI/.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

High-fidelity GAN Inversion with Padding Space (2022, methodology): “With the start-of-the-art synthesis performance, they are widely used for GAN inversion studies [1, 2, 22,36,37,42,43,47,54,57].”
InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation (2023, background): “By properly exploiting their latent structure, prior works succeeded in image editing [33, 79, 80, 81, 82, 83, 84], semantic control [55, 56, 85], disentangled control direction discovery [86, 87, 88, 89], etc..”
Re-Imagen: Retrieval-Augmented Text-to-Image Generator (2022, methodology): “Previously, GANs (Goodfellow et al., 2014) have been used to achieve signiﬁcant performance on image editing (Zhu et al., 2016; Abdal et al., 2019; Zhu et al., 2020; Roich et al., 2021; Tov et al., 2021; Wang et al., 2022; Alaluf et al., 2022).”
Third Time's the Charm? Image and Video Editing with StyleGAN3 (2022, methodology): “Inversion methods typically directly optimize the latent vector to minimize the reconstruction error of a given image [1, 2, 10, 17, 24, 43, 77, 81, 86], train an encoder to map an image to its latent representation [6, 25, 31, 38, 45, 49, 51, 55, 67, 71], or design a hybrid approach combining both.”
Pretraining is All You Need for Image-to-Image Translation (2022, background): “There are prior attempts [37, 9, 56, 49, 51, 33, 31] to leverage a pretrained model as generative prior for conditional image synthesis, image editing and restoration.”
DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation (2021, methodology): “To demonstrate the nearly perfect reconstruction performance of our method, we perform the quantitative comparison with SOTA GAN inversion methods, pSp [45], e4e [55], ReStyle [3] and HFGI [58].”
Self-supervised Learning of Adversarial Example: Towards Good Generalizations for Deepfake Detection (2022, background): “The realistic image generation brought by the generative adversarial network (GAN) raises a security issue that human portraits can be easily substituted to provide malicious bioinformatics [11, 45, 15, 44, 51, 56, 40, 54, 48].”
Prompt-to-Prompt Image Editing with Cross Attention Control (2022, background): “This process, known as inversion , has recently drawn considerable attention for GANs, e.g., [51, 1, 3, 35, 50, 43, 45, 47], but has not yet been fully addressed for text-guided diffusion models.”
DiffEdit: Diffusion-based semantic image editing with mask guidance (2022, methodology): “Wang et al. (2022b) use DDIM encoding of the input image, and then decode on edited sketches or semantic segmentation maps.”
HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing (2021, methodology): “Generally, inversion methods either directly optimize the latent vector to minimize the error for a given image [1, 2, 7, 15, 21, 41, 81, 85, 86], train an encoder over a large number of samples to learn a mapping from an image to its latent representation [5, 22, 22, 31, 37, 44, 52, 53, 57, 68, 74]”