Neural Photo Editing with Introspective Adversarial Networks

By Andrew Brock, Theodore Lim, James M. Ritchie, Nick Weston. 2016

477 citations recorded by the source corpus.

The increasingly photorealistic sample quality of generative image models suggests their feasibility in applications beyond image generation. We present the Neural Photo Editor, an interface that leverages the power of generative neural networks to make large, semantically coherent changes to existing images. To tackle the challenge of achieving accurate reconstructions without loss of feature quality, we introduce the Introspective Adversarial Network, a novel hybridization of the VAE and GAN. Our model efficiently captures long-range dependencies through use of a computational block based on weight-shared dilated convolutions, and improves generalization performance with Orthogonal Regularization, a novel weight regularization method. We validate our contributions on CelebA, SVHN, and CIFAR-100, and produce samples and reconstructions with high visual fidelity.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Large Scale GAN Training for High Fidelity Natural Image Synthesis (2018, methodology): “Across runs in Table 1, we observe that without Orthogonal Regularization, only 16% of models are amenable to truncation, compared to 60% when trained with Orthogonal Regularization.”
Spectral Normalization for Generative Adversarial Networks (2018, methodology): “For orthonormal , we initialized the each weight of D with a randomly selected orthonormal operator and trained GANs with the objective function augmented with the regularization term used in Brock et al. (2016).”
Generative Adversarial Networks (2021, neutral): “Researchers are also looking into techniques to reduce the quantity of data necessary for GAN training, also known as semi-supervised learning, or to enrich the Training Data with generative models.”
D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, background): “the GAN training procedure can be unstable [9, 8, 63], lack a informative objective for measuring progress [4], and struggle with discrete data [102].”
Diffusion Models Beat GANs on Image Synthesis (2021, background): “Furthermore, GANs are often difficult to train, collapsing without carefully selected hyperparameters and regularizers [5, 35, 4].”
Diagnosing and Enhancing VAE Models (2019, background): “, 2018), interpretable encoder/inference network (Brock et al., 2016), outlier-robustness (Dai et al.”
StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation (2017, methodology): “The idea of conditional image generation has also been successfully applied to domain transfer [8, 27], superresolution imaging[13], and photo editing [2, 26].”
StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks (2016, background): “There is also work conditioned on images to generate images, including photo editing [2, 39], domain transfer [32, 12] and super-resolution [31, 15].”
Adversarially Learned Inference (2016, background): “For instance, GANs don’t allow the sort of neural photo manipulations showcased in (Brock et al., 2016).”
NIPS 2016 Tutorial: Generative Adversarial Networks (2016, methodology): “models that create images. A video demonstration of iGAN is available at the following URL: https://www.youtube.com/watch?v=9c4z6YsBGQ0 6 (Goodfellow 2016) Introspective Adversarial Networks youtube (Brock et al 2016) Figure 6: Brock et al. (2016) developed introspective adversarial networks (IAN). ”
StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks (2017, background): “There are also works conditioned on images to generate images, including photo editing [4], [57], domain transfer [18], [43] and super-resolution [23], [42].”
Unsupervised Cross-Domain Image Generation (2016, methodology): “This is modeled as a ternary discriminative function D (Reed et al., 2016; Brock et al., 2016).”
High Fidelity Speech Synthesis with Adversarial Networks (2019, neutral): “…(Miyato et al., 2018) and orthogonal initialisation (Saxe et al., 2014) in both the generator and discriminator(s), and apply off-diagonal orthogonal regularisation (Brock et al., 2016; 2019) and exponential moving averaging to the generator weights with a decay rate of 0.9999 for sampling.”
Fader Networks: Manipulating Images by Sliding Attributes (2017, background): “The neural photo editor [3] disentangles factors of variations in natural images for image edition.”
Controllable Text-to-Image Generation (2019, methodology): “[2] introduced a hybrid model using VAEs [9] and GANs, which achieved an accurate reconstruction without loss of image quality.”
Scribbler: Controlling Deep Image Synthesis with Sketch and Color (2016, background): “for image editing [4, 56] or network visualization [31, 32]), the space itself is not semantically well organized – the particular dimensions of the latent vector do not correspond to semantic attributes although mapping them to an intermediate structure image [47] can give us more insight.”
Semantic Image Synthesis via Adversarial Learning (2017, background): “On the other hand, there are many proposed approaches [17, 35, 11, 42, 18, 40, 1, 37, 19, 12] that condition GAN on images to synthesize images for various tasks.”