Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks

By Chuan Li, Michael Wand. 2016

1,522 citations recorded by the source corpus.

This paper proposes Markovian Generative Adversarial Networks (MGANs), a method for training generative networks for efficient texture synthesis. While deep neural network approaches have recently demonstrated remarkable results in terms of synthesis quality, they still come at considerable computational costs (minutes of run-time for low-res images). Our paper addresses this efficiency issue. Instead of a numerical deconvolution in previous work, we precompute a feed-forward, strided convolutional network that captures the feature statistics of Markovian patches and is able to directly generate outputs of arbitrary dimensions. Such network can directly decode brown noise to realistic texture, or photos to artistic paintings. With adversarial training, we obtain quality comparable to recent neural texture synthesis methods. As no optimization is required at generation time, our run-time performance (0.25 M pixel images at 25 Hz) surpasses previous neural texture synthesizers by a significant margin (at least 500 times faster). We apply this idea to texture synthesis, style transfer, and video stylization.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Free-Form Image Inpainting With Gated Convolution (2018, methodology): “the feature statistics of Markovian patches [21].”
Image-to-Image Translation with Conditional Adversarial Networks (2016, methodology): “A similar PatchGAN architecture was previously proposed in [35], for the purpose of capturing local style statistics.”
Local Aggregation for Unsupervised Learning of Visual Embeddings (2019, background): “Although the features learned by generative models have been put to a wide variety of exciting uses [44, 74, 13, 47, 37], their power as latent representations for downstream visual tasks such as object recognition has yet to be fully realized.”
StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation (2017, methodology): “We leverage PatchGANs [7, 15, 33] for the discriminator network, which classifies whether local image patches are real or fake.”
Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks (2017, methodology): “For the discriminator networks we use 70 × 70 PatchGANs [22, 30, 29], which aim to classify whether 70× 70 overlapping image patches are real or fake.”
Learning from Simulated and Unsupervised Images through Adversarial Training (2016, background): “Li and Wand [19] propose a Markovian GAN for efficient texture synthesis.”
Perceptual Losses for Real-Time Style Transfer and Super-Resolution (2016, background): “Concurrent with our work, [26,27] also propose feed-forward approaches for fast style transfer.”
A Learned Representation For Artistic Style (2016, methodology): “Ulyanov et al. (2016a), Li & Wand (2016) and Johnson et al. (2016) tackle this problem by introducing a feedforward style transfer network , which is trained to go from content to pastiche image in one pass.”
Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization (2017, methodology): “GANs have also been applied to style transfer [31] and cross-domain image generation [50, 3, 23, 38, 37, 25].”
Mode Regularized Generative Adversarial Networks (2016, methodology): “…image synthesis from text Reed et al. (2016) and edge map Isola et al. (2016), real-time image manipulation Zhu et al. (2016), temporal image generation Zhou & Berg (2016); Saito & Matsumoto (2016); Vondrick et al. (2016), texture synthesis, style transfer, and video stylization Li & Wand (2016).”
Recent Progress on Generative Adversarial Networks (GANs): A Survey (2019, methodology): “Based on GANs, Li and Wand [53] proposed a texture synthesis approach, which is named Markovian Generative Adversarial Networks (MGAN).”
Shape Completion Using 3D-Encoder-Predictor CNNs and Shape Synthesis (2016, background): “Recently deep learning also explore models for generative tasks; for instance, with generative adversarial networks (GANs) [10, 19, 18, 44].”
Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling (2016, background): “To name a few, Wang and Gupta [2016] discussed how to model image style and structure with sequential GANs, Li and Wand [2016] and Zhu et al. [2016] used GAN for texture synthesis and image editing, respectively, and Im et al. [2016] z Figure 1: The generator in 3D-GAN.”
High Fidelity Speech Synthesis with Adversarial Networks (2019, neutral): “Discriminators operating on windows of the input have been used in adversarial texture synthesis (Li & Wand, 2016) and image translation (Isola et al., 2017; Zhu et al., 2017).”
DualGAN: Unsupervised Dual Learning for Image-to-Image Translation (2017, methodology): “Conventionally, these tasks have been tackled separately due to their intrinsic disparities [7, 8, 21, 12, 4, 18].”
Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space? (2019, neutral): “texture synthesis [18, 33, 28], video generation [31, 30], image-to-image translation [11, 36, 1, 24] and object detection [19].”
Fader Networks: Manipulating Images by Sliding Attributes (2017, methodology): “The exact choice of the reconstruction loss is not fundamental in our approach, and adversarial losses such as PatchGAN [13] could be used in addition to the MSE at this stage to obtain better textures or sharper images, as in [10].”
Exploring the structure of a real-time, arbitrary neural artistic stylization network (2017, background): “, style transfer network, to explicitly learn the transformation from a photograph a particular painting style [14, 16, 25].”
Scribbler: Controlling Deep Image Synthesis with Sketch and Color (2016, methodology): “Conditional GANs have also been used to transform images into different domains such as product images [52] or different artistic styles [27].”