Texture Synthesis with Spatial Generative Adversarial Networks

By Nikolay Jetchev, Urs M. Bergmann, Roland Vollgraf. 2016

226 citations recorded by the source corpus.

Generative adversarial networks (GANs) are a recent approach to train generative models of data, which have been shown to work particularly well on image data. In the current paper we introduce a new model for texture synthesis based on GAN learning. By extending the input noise distribution space from a single vector to a whole spatial tensor, we create an architecture with properties well suited to the task of texture synthesis, which we call spatial GAN (SGAN). To our knowledge, this is the first successful completely data-driven texture synthesis method based on GANs. Our method has the following features which make it a state of the art algorithm for texture synthesis: high image quality of the generated textures, very high scalability w.r.t. the output texture size, fast real-time forward generation, the ability to fuse multiple diverse source images in complex textures. To illustrate these capabilities we present multiple experiments with different classes of texture images and use cases. We also discuss some limitations of our method with respect to the types of texture images it can synthesize, and compare it to other neural techniques for texture generation.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Generative Adversarial Networks (2021, neutral): “It was proved successful in many domains such as computer vision [14, 29, 25, 38], semantic segmentation [39, 27, 70, 24], time-series synthesis [9, 23], image editing [61, 36, 19, 3, 75], natural language processing [15, 28, 22], text-to-image generation [59, 58, 54], and many more.”
Local Aggregation for Unsupervised Learning of Visual Embeddings (2019, background): “Although the features learned by generative models have been put to a wide variety of exciting uses [44, 74, 13, 47, 37], their power as latent representations for downstream visual tasks such as object recognition has yet to be fully realized.”
Recent Progress on Generative Adversarial Networks (GANs): A Survey (2019, background): “Jetchev et al. [54] proposed a Spatial GAN(SGAN), which was the ﬁrst application of GANs with fully unsupervised learning in texture synthesis.”
SinGAN: Learning a Generative Model From a Single Natural Image (2019, background): “Unconditional single image GANs have been explored only in the context of texture generation [3, 27, 31].”
Image De-Raining Using a Conditional Generative Adversarial Network (2017, background): “[31] proposed spatial GANs for texture synthesis.”
Generative Adversarial Networks: A Survey and Taxonomy (2019, methodology): “GANs have been applied to various domains such as computer vision [7]–[14], natural language processing [15]–[18], time series synthesis [19]–[23], semantic segmentation [24]–[28] etc. GANs belong to the family of generative models.”
Learning Texture Manifolds with the Periodic Spatial GAN (2017, methodology): “Receptive ﬁeld and image patch size can both affect learning (Jetchev et al., 2016).”
Training‐Image Based Geostatistical Inversion Using a Spatial Generative Adversarial Neural Network (2017, methodology): “For all our experiments, we used squared or cubic ﬁlters with side length ( N i in equation (2)) in the range 3-9 depending on the case study and the same activation functions ( f ( · ) in equation (2)) as in Jetchev et al. (2016).”
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, methodology): “The discriminator and generator loss functions of the Relativistic Standard GAN (RSGAN) is: LRSGAND = −E(xr,xg ) [log (sigmoid (C (xr)− C (xg)))] , (51) LRSGANG = −E(xr,xg ) [log (sigmoid (C (xg)− C (xr)))] .”
Auto-Embedding Generative Adversarial Networks For High Resolution Image Synthesis (2019, background): “neural networks (DNNs) in many real-world applications, such as texture synthesis [11], [20], [32], [41], super-resolution [15], [30], [50] and attribute editing [9], [16], [55].”
A Survey on Generative Adversarial Networks: Variants, Applications, and Training (2020, background): “A newly introduced least square loss in LS-GAN not only classifies the actual and fake samples but also brings the fake data nearer to the real data for better architecture and real-world generation of data.”