Texture synthesis and the controlled generation of natural stimuli using convolutional neural networks

By Leon A. Gatys, Alexander S. Ecker, Matthias Bethge. 2015

132 citations recorded by the source corpus.

It is a long standing question how biological systems transform visual inputs to robustly infer high-level visual information. Research in the last decades has established that much of the underlying computations take place in a hierarchical fashion along the ventral visual pathway. However, the exact processing stages along this hierarchy are difficult to characterise. Here we present a method to generate stimuli that will allow a principled description of the processing stages along the ventral stream. We introduce a new parametric texture model based on the powerful feature spaces of convolutional neural networks optimised for object recognition. We show that constraining a spatial summary statistic over feature maps suffices to synthesise high-quality natural textures. Moreover we establish that our texture representations continuously disentangle high-level visual information and demonstrate that the hierarchical parameterisation of the texture model naturally enables us to generate novel types of stimuli for systematically probing mid-level vision.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Image-to-Image Translation with Conditional Adversarial Networks (2016, neutral): “This connection was previously explored in [35], and is also the common assumption in models of texture [15, 19] and style [14, 23, 20, 34].”
A note on the evaluation of generative models (2015, background): “It is therefore unsurprising that other approaches than density estimation are much more effective for image synthesis (Portilla & Simoncelli, 2000; Dosovitskiy et al., 2015; Gatys et al., 2015).”
A Neural Algorithm of Artistic Style (2015, methodology): “When Convolutional Neural Networks are trained on object recognition, they develop a representation of the image that makes object information increasingly explicit along the processing hierarchy.(8) Therefore, along the processing hierarchy of the network, the input image is transformed into repres”
Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks (2016, background): “The astonishing complexity of the obtained visual patterns has immediately spurred hope for new generative models: Gatys et al. [8,7] drove deconvolution by global covariance statistics of feature vectors on higher network layers, obtaining unprecedented results in artistic style transfer.”
Bilinear CNN Models for Fine-Grained Visual Recognition (2015, background): “[18], [19] showed that the Gram matrix representation constructed from multiple layers of a CNN is an effective texture representation and can be used to generate new textures or stylize images.”
Visualizing Deep Convolutional Neural Networks Using Natural Pre-images (2015, methodology): “[12, 13] have used the L-BFGSB algorithm [54] to optimize their texture/style loss (8).”
Understanding deep convolutional networks (2016, background): “Impressive texture synthesis results have been obtained with deep convolutional networks trained on image data bases [14], but with much more output coefficients.”
Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis (2016, methodology): “Concretely, we implement this by splitting up the input into a “style” image and a “content” image [8, 12].”
Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet (2019, methodology): “For VGG-16 an algorithm that comes close to this objective is the popular texture synthesis algorithm based on the Gram features of the hidden layer activations (Gatys et al., 2015), Figure 5.”
Discriminative Regularization for Generative Models (2016, methodology): “In this effort we are heavily inspired by recently introduced texture synthesis method of (Gatys et al., 2015b) as well as the DeepStyle model of (Gatys et al., 2015a).”
High-Resolution Image Inpainting Using Multi-scale Neural Patch Synthesis (2016, background): “Style Transfer In order to create realistic image textures, our work is motivated by the recent success of neural style transfer [15, 16, 28, 3, 39, 22].”
Super-Resolution with Deep Convolutional Sufficient Statistics (2015, methodology): “Our approach also relates to the method proposed by Gatys et al. (2015) for texture synthesis.”
Synthesizing Images of Humans in Unseen Poses (2018, background): “Initializing the foreground with noise provides high-frequency gradients useful for texture synthesis [3].”
Kernel Pooling for Convolutional Neural Networks (2017, background): “Representative examples include fine-grained visual recognition [23, 11], visual question answering [10], texture representation and synthesis [13, 22], face recognition [35] and style transfer [12].”
Visualizing and Understanding Deep Texture Representations (2015, methodology): “[13] propose a method for texture synthesis from an input image I by obtaining an image x ∈ RH×W×C that matches the bilinear features at various layers by solving the following optimization: min x n ∑”
A Style-Aware Content Loss for Real-time HD Style Transfer (2018, methodology): “[13,12] proposed a more flexible iterative optimization approach based on a pre-trained VGG19 network [41].”
CartoonGAN: Generative Adversarial Networks for Photo Cartoonization (2018, methodology): “They use the feature maps of a pre-trained VGG network to represent the content and optimize the result image, such that it retains the content from the content image while matching the texture information of the style image, where the texture is described using the global Gram matrix [7].”
Deep Spatial-Semantic Attention for Fine-Grained Sketch-Based Image Retrieval (2017, background): “However, recent efforts [6, 22] on visualising what each layer of a CNN actually learns”
A Powerful Generative Model Using Random Weights for the Deep Image Representation (2016, methodology): “[8, 12] define a squared loss on the correlations between feature maps of some layers and synthesize natural textures of high perceptual quality using the pretrained CNN called VGG [3].”