Generative Visual Manipulation on the Natural Image Manifold

By Jun-Yan Zhu, Philipp Krahenbuhl, Eli Shechtman, Alexei A. Efros. 2016

1,449 citations recorded by the source corpus.

Realistic image manipulation is challenging because it requires modifying the image appearance in a user-controlled way, while preserving the realism of the result. Unless the user has considerable artistic skill, it is easy to “fall off” the manifold of natural images while editing. In this paper, we propose to learn the natural image manifold directly from data using a generative adversarial neural network. We then define a class of image editing operations, and constrain their output to lie on that learned manifold at all times. The model automatically adjusts the output keeping all edits as realistic as possible. All our manipulations are expressed in terms of constrained optimization and are applied in near-real time. We evaluate our algorithm on the task of realistic photo manipulation of shape and color. The presented method can further be used for changing one image to look like the other, as well as generating novel imagery from scratch based on user’s scribbles.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Media Technology, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Media Technology

In Generative Models

Citations in the corpus

On Buggy Resizing Libraries and Surprising Subtleties in FID Calculation (2021, background): “A wide range of image and video synthesis applications [45, 54, 67, 83] have been enabled, as a result of tremendous progress in deep generative models such as GANs [9, 28, 34, 37, 59], VAEs [41, 55, 61], autoregressive models [52], flow-based models [18, 40], and energy-based models [21, 50, 63].”
Image-to-Image Translation with Conditional Adversarial Networks (2016, methodology): “These papers have achieved impressive results on inpainting [40], future state prediction [60], image manipulation guided by user constraints [61], style transfer [35], and superresolution [33].”
Unrolled Generative Adversarial Networks (2016, background): “, 2016) and has been applied to explore the visual manifold of GANs in (Zhu et al., 2016).”
GAN Inversion: A Survey (2021, methodology): “Recent improvements of optimization-based methods emphasize on how to ﬁnd the desired latent code more quickly thus propose several initialization strategies [21], [22] and optimizers [20], [24].”
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs (2017, methodology): “[62] developed an optimization method for editing the object appearance based on the priors learned by GANs.”
Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks (2017, background): “Generative Adversarial Networks (GANs) [14, 58] have achieved impressive results in image generation [5, 35], image editing [61], and representation learning [35, 39, 33].”
Learning from Simulated and Unsupervised Images through Adversarial Training (2016, background): “The recently proposed iGAN [49] enables users to change the image interactively on a natural image manifold.”
On the Quantitative Analysis of Decoder-Based Generative Models (2016, background): “…models have dramatically pushed forward the state-of-the-art in generative modelling by generating convincing samples of images (Radford et al., 2016), achieving state-of-the-art semi-supervised learning results (Salimans et al., 2016), and enabling automatic image manipulation (Zhu et al., 2016).”
Semantic Image Inpainting with Deep Generative Models (2016, methodology): “With some user interaction, GANs have been applied in interactive image editing [40].”
StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks (2016, background): “There is also work conditioned on images to generate images, including photo editing [2, 39], domain transfer [32, 12] and super-resolution [31, 15].”
NIPS 2016 Tutorial: Generative Adversarial Networks (2016, background): “(Goodfellow 2016) iGAN youtube (Zhu et al 2016) Figure 5: Zhu et al. (2016) developed an interactive application called interactive generative adversarial networks (iGAN).”
StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks (2017, background): “There is also work conditioned on images to generate images, including photo editing [3, 52], domain transfer [39, 15] and super-resolution [38, 19].”
Toward Controlled Generation of Text (2017, background): “Despite their impressive advances in visual domain, such as image generation (Radford et al., 2015), learning interpretable image representations (Chen et al., 2016), and image editing (Zhu et al., 2016), applications to natural language generation have been relatively less studied.”
Neural Photo Editing with Introspective Adversarial Networks (2016, methodology): “The method of iGAN (Zhu et al., 2016) bears the most relation to our interface.”
Multimodal Unsupervised Image-to-Image Translation (2018, background): “[81], which contain images of shoes and handbags with edge maps generated by HED [82].”
Photographic Image Synthesis with Cascaded Refinement Networks (2017, background): “Researchers have also studied image inpainting [32], superresolution [3, 17, 19], novel view synthesis [10, 45, 51], and interactive image manipulation [52].”