Swapping Autoencoder for Deep Image Manipulation

By Taesung Park, Jun-Yan Zhu, Oliver Wang, Jingwan Lu, Eli Shechtman, Alexei A. Efros, Richard Zhang. 2020

373 citations recorded by the source corpus.

Deep generative models have become increasingly effective at producing realistic images from randomly sampled seeds, but using such models for controllable manipulation of existing images remains challenging. We propose the Swapping Autoencoder, a deep model designed specifically for image manipulation, rather than random sampling. The key idea is to encode an image with two independent components and enforce that any swapped combination maps to a realistic image. In particular, we encourage the components to represent structure and texture, by enforcing one component to encode co-occurrent patch statistics across different parts of an image. As our method is trained with an encoder, finding the latent codes for a new input image becomes trivial, rather than cumbersome. As a result, it can be used to manipulate real input images in various ways, including texture swapping, local and global editing, and latent code vector arithmetic. Experiments on multiple datasets show that our model produces better results and is substantially more efficient compared to recent generative models.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, background): “These properties c include (but are not limited to2) labels [61], text descriptions [60, 76], noisy or partial observations [11, 5, 47, 24], and manipulation constraints [69].”
Alias-Free Generative Adversarial Networks (2021, background): “They have been used for a variety of applications, including image editing [47, 53, 42, 21, 38, 3], domain translation [66, 36, 58, 41], and video generation [55, 15, 23].”
Disentangling speech from surroundings in a neural audio codec (2022, background): “Disentangled representations can be learned by partitioning feature vectors in the latent space and mixing the partitions while training the model to produce realistic decoded outputs, thereby capturing one factor of variation in each partition [20] or to synthesize new samples that share attributes”
Image-to-Image Translation: Methods and Applications (2021, background): “style transfer artistic:[113], [1] artistic:[15] artistic:[2], [36], [136], [137], [145]; photo-realistic: [102], [139], [2], [75], [136], [137], artistic:[152]; photo-realistic: [90], [76], [77], [77], [143], [152], [154] \ artistic:[155], [81], [166]; photo-realistic: [155], [156], [81] artistic:[”
One-Step Diffusion with Distribution Matching Distillation (2023, background): “…them, GANs have shown unprecedented quality in realism [4, 26– 28, 30, 67], particularly when the GAN loss can be combined with task-speciﬁc, auxiliary regression losses to mitigate training instability, ranging from paired image translation [24, 54, 79, 89] to unpaired image editing [37, 55, 94].”
Third Time's the Charm? Image and Video Editing with StyleGAN3 (2022, background): “Specifically, StyleGAN generators [33–36] synthesize exceed-ingly realistic images, and enable editing [4,5,9,15,42,48, 83], and image-to-image translation [46,47,55], particularly in well-structured domains.”
Text2LIVE: Text-Driven Layered Image and Video Editing (2022, methodology): “We adopt the Two-alternative Forced Choice (2AFC) protocol suggested in [18,29].”
BlobGAN: Spatially Disentangled Scene Representations (2022, background): “faces and cars, or changing textures for scenes [58].”
EditGAN: High-Precision Semantic Image Editing (2021, background): “(vi) Finally, another line of research targets primarily very high-level image and photo stylization and global appearance modiﬁcations [37, 41, 46, 52–57].”
VITON-HD: High-Resolution Virtual Try-On via Misalignment-Aware Normalization (2021, background): “Since the emergence of pix2pix [14], numerous cGANs conditioned on input images have been proposed to generate high-resolution images in a stable manner [32, 1, 21].”
ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models (2021, background): “…as generative adversarial networks (GAN) [3, 10, 19], normalizing flows [21], and variational autoencoders [42], have shown remarkable quality in image generation, and have been applied to numerous purposes such as image-to-image translation [7, 11, 31, 32, 35, 47] and image editing [1, 12, 36].”
Ensembling with Deep Generative Views (2021, background): “Moreover, recent alternative architectures trained specifically for efficiency [30] or invertibility [38] may help further reduce the computational cost of image reconstruction.”
Language-Driven Image Style Transfer (2021, background): “Photorealistic style transfer [57, 58, 59, 60, 61, 62] aims at applying reference styles on scenes without hurting details and satisfying contradictory objectives.”
Editing Conditional Radiance Fields (2021, background): “Recent works have focused on integrating user interactions into deep networks either through optimization [1, 9, 81, 10] or a feed-forward network with user-guided inputs [79, 57, 52, 49].”
Imagic: Text-Based Real Image Editing with Diffusion Models (2022, methodology): “Users were asked to choose between the left result and the right one, akin to the standard practice of Two-Alternative Forced Choice (2AFC) [8, 35, 42].”