Texture Networks: Feed-forward Synthesis of Textures and Stylized Images

By Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, Victor S. Lempitsky. 2016

1,012 citations recorded by the source corpus.

Gatys et al. recently demonstrated that deep networks can generate beautiful textures and stylized images from a single texture example. However, their methods require a slow and memory-consuming optimization process. We propose here an alternative approach that moves the computational burden to a learning stage. Given a single example of a texture, our approach trains compact feed-forward convolutional networks to generate multiple samples of the same texture of arbitrary size and to transfer artistic style from a given image to any other image. The resulting networks are remarkably light-weight and can generate textures of quality comparable to Gatys et al., but hundreds of times faster. More generally, our approach highlights the power and flexibility of generative feed-forward models trained with complex and expressive loss functions.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks (2017, neutral): “Neural Style Transfer [11, 21, 48, 10] is another way to perform image-to-image translation, which synthesizes a novel image by combining the content of one image with the style of another image (typically a painting) by matching the Gram matrix statistics of pre-trained deep features.”
Instance Normalization: The Missing Ingredient for Fast Stylization (2016, methodology): “Abstract It this paper we revisit the fast stylization method introduced in Ulyanov et al. (2016). We show how a small change in the stylization architecture results in a significant qualitative improvement in the generated images.”
A Learned Representation For Artistic Style (2016, methodology): “Building off the instance normalization technique proposed in Ulyanov et al. (2016b), we augment the γ and β parameters so that they’re N × C matrices, where N is the number of styles being modeled and C is the number of output feature maps.”
Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization (2017, methodology): “The optimization process is replaced by feed-forward neural networks in [24, 51, 52].”
Unsupervised Cross-Domain Image Generation (2016, methodology): “While initially style transfer was obtained by a slow optimization process (Gatys et al., 2016), recently, the emphasis was put on feed-forward methods (Ulyanov et al., 2016; Johnson et al., 2016).”
Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks (2016, methodology): “For example the eyes in [32]’s result and the entire face in [8]’s result are not textured.”
EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis (2016, methodology): “, a given painting style) onto other images [24, 53], however a single network again only produces a single texture, and textures in all input images are replaced by the single style that the network has been trained for.”
Adversarial Transformation Networks: Learning to Generate Adversarial Examples (2017, methodology): “Ulyanov et al. (2016) removed the need for a gradient descent procedure to operate on every source image to which a new artistic style was to be applied, and replaced it with a single forward pass through a separate network.”
Improved Texture Networks: Maximizing Quality and Diversity in Feed-Forward Stylization and Texture Synthesis (2017, methodology): “In particular, [19] introduced texture networks to generate textures of a certain kind, as in [4], or to apply”
Exploring the structure of a real-time, arbitrary neural artistic stylization network (2017, background): “, style transfer network, to explicitly learn the transformation from a photograph a particular painting style [14, 16, 25].”
High-Resolution Image Inpainting Using Multi-scale Neural Patch Synthesis (2016, background): “Style Transfer In order to create realistic image textures, our work is motivated by the recent success of neural style transfer [15, 16, 28, 3, 39, 22].”
Fast Patch-based Style Transfer of Arbitrary Style (2016, methodology): “Compared to existing feedforward approaches [6, 17, 33, 34], the biggest advantage of our feedforward procedure is the ability to use new style images with only a single trained inverse network.”
Universal Style Transfer via Feature Transforms (2017, methodology): “[15] TNet [27] DeepArt [9] Ours Arbitrary √ √ × √ √ Efficient √ √ √ × √ Learning-free × × × √ √”
DA-GAN: Instance-Level Image Translation by Deep Attention Generative Adversarial Networks (2018, background): “Neural Style Transfer [8] [15, 36] is another way to perform image-to-image translation, which synthesizes a image by combining the content of one image with the style of another image based on pre-trained deep features.”
Regularization for Deep Learning: A Taxonomy (2017, background): “A given input x can have an associated target t which dictates the desired output f(x) directly (or in some applications indirectly (Ulyanov et al., 2016; Johnson et al., 2016)).”
Dynamic Instance Normalization for Arbitrary Style Transfer (2019, background): “(Johnson, Alahi, and Fei-Fei 2016; Ulyanov et al. 2016; Li and Wand 2016).”