TextureGAN: Controlling Deep Image Synthesis with Texture Patches

By Wenqi Xian, Patsorn Sangkloy, Jingwan Lu, Chen Fang, Fisher Yu, James Hays. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017

300 citations recorded by the source corpus.

In this paper, we investigate deep image synthesis guided by sketch, color, and texture. Previous image synthesis methods can be controlled by sketch and color strokes but we are the first to examine texture control. We allow a user to place a texture patch on a sketch at arbitrary locations and scales to control the desired output texture. Our generative network learns to synthesize objects consistent with these texture suggestions. To achieve this, we develop a local texture loss in addition to adversarial and content loss to train the generative network. We conduct experiments using sketches generated from real images and textures sampled from a separate texture database and results show that our proposed algorithm is able to generate plausible images that are faithful to user controls. Ablation studies show that our proposed pipeline can generate more realistic images than adapting existing methods directly.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/ad527d3375ffd1d7f114011541bdcf76d7abcb47

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space? (2019, unknown): “texture synthesis [18, 33, 28], video generation [31, 30], image-to-image translation [11, 36, 1, 24] and object detection [19].”
A Survey on Generative Adversarial Networks: Variants, Applications, and Training (2020, methodology): “…transfer [94 - 96], facial landmark d 1 etection [97, 98], image super-resolution [99 - 102], texture synthesis [103, 105, 106], sketch synthesis [107, 110, 111], image-to-image translation [34, 112 - 120], face frontal view generation [121-123, Although GANs have achieved some incredible…”
Attribute Manipulation Generative Adversarial Networks for Fashion Images (2019, unknown): “Generative networks have also been widely applied on various fashion-related tasks such as virtual try-on [15, 12, 30, 26] and fashion design/generation [19, 31, 29].”
Texture Fields: Learning Texture Representations in Function Space (2019, background): “Recent work has shown that image generation methods [3, 11, 19, 20, 24, 30, 47] can be improved by exploiting 3D knowledge about the generated shapes [1, 2, 49].”
Learning a Neural 3D Texture Space From 2D Exemplars (2019, background): “At any rate, none of the texture works in graphics or vision [17, 5, 23, 4, 2, 26, 27] generate a space of textures, such as we suggest here, but all work on a single texture while the ones that work on a space of exemplars [29, 16] do not create stochastic textures.”
SDEdit: Image Synthesis and Editing with Stochastic Differential Equations (2021, background): “Conditional GANs [26, 72] for image editing learn to directly generate an image based on a user input, and have demonstrated success on a variety of tasks including image synthesis and editing [42, 11, 12, 59, 39, 73], sketch2photo [47], inpainting [40, 25, 65, 35], photo colorization [67, 34, 68, 2”
SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations (2021, background): “, 2018), semantic image texture and geometry synthesis (Zhou et al., 2018; Guérin et al., 2017; Xian et al., 2018).”
Layout2image: Image Generation from Layout (2020, background): “…Pathak et al. 2016; Yang et al. 2017; Zhao et al. 2018; Zhu et al. 2017a,b), sketches (Isola et al. 2017; Sangkloy et al. 2017; Wang et al. 2018; Xian et al. 2018; Zhu et al. 2017b), scene graphs (Johnson et al. 2018), dialogues (Kim et al. 2017; Sharma et al.…”
Context-aware Feature Generation For Zero-shot Semantic Segmentation (2020, background): “embedding, known as mode collapse problem [42, 51]; 2) Although ZS3Net (GC) utilizes relational graphs to encode spatial object arrangement, the contextual cues it considers are object-level and only limited to spatial object arrangement.”
Image2StyleGAN++: How to Edit the Embedded Images? (2019, background): “object detection [23], texture synthesis [22, 37, 31], image-to-image translation [16, 42, 28, 25] and video generation [33, 32, 35, 34].”
Language-Driven Image Style Transfer (2021, methodology): “Inspired by texture synthesis [107, 108], images with analogous patch patterns should appear perceptually similar texture patterns.”
Toward Multimodal Image-to-Image Translation (2017, background): “However, the quality has been attained at the expense of multimodality, as the generator learns to largely ignore the random noise vector when conditioned on a relevant context [20, 34, 40, 45, 47, 55].”
Deep Learning for Free-Hand Sketch: A Survey (2020, background): “Sketch generation can be solved in various frameworks, e . g ., variational autoencoder (VAE) [16], [59], [97], Generative Adversarial Network (GAN) [98], VAE-GAN [98], reinforcement learning (RL) [98], [99].”
Swapping Autoencoder for Deep Image Manipulation (2020, methodology): “Similar ideas for modeling co-occurences have been used for propagating a single texture in a supervised setting [74], self-supervised representation learning [33], and identifying image composites [29].”
Stochastic Conditional Generative Networks with Basis Decomposition (2019, background): “23] to variational autoencoders [13]; in particular variants with conditions [17, 24, 25] show promising results. With the great power of GANs [7], conditional generative adversarial networks (cGANs) [12, 18, 21, 27, 29, 31] achieve great progress on visually appealing images given conditions. Howev”
InfinityGAN: Towards Infinite-Resolution Image Synthesis (2021, unknown): “Repetitive texture synthesis methods [5, 39] generalize to high resolutions.”