Controlling Perceptual Factors in Neural Style Transfer

By Leon A. Gatys, Alexander S. Ecker, Matthias Bethge, Aaron Hertzmann, Eli Shechtman. Computer Vision and Pattern Recognition, 2016

498 citations recorded by the source corpus.

Neural Style Transfer has shown very exciting results enabling new forms of image manipulation. Here we extend the existing method to introduce control over spatial location, colour information and across spatial scale. We demonstrate how this enhances the method by allowing high-resolution controlled stylisation and helps to alleviate common failure cases such as applying ground textures to sky regions. Furthermore, by decomposing style into these perceptual factors we enable the combination of style information from multiple sources to generate new, perceptually appealing styles from existing ones. We also describe how these methods can be used to more efficiently produce large size, high-quality stylisation. Finally we show how the introduced control measures can be applied in recent methods for Fast Neural Style Transfer.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/acc529f6f65fbdbae1aff14682168b5132143f28

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

A Learned Representation For Artistic Style (2016, methodology): “Subsequent work has brought some performance improvements to style transfer networks, e.g. with respect to color preservation (Gatys et al., 2016a) or style transfer quality (Ulyanov et al., 2016b), but to our knowledge the problem of the single-purpose nature of style transfer networks remains…”
Exploring the structure of a real-time, arbitrary neural artistic stylization network (2017, background): “Pairing a secondary constraint to preserve the content of an image – as measured by the higher level layers of the same image classification network – extended this idea to artistic style transfer [9] (see also [10]).”
Rethinking and Improving the Robustness of Image Style Transfer (2021, methodology): “A consistent observation from this work [43, 24, 30, 23, 25, 19, 9, 2, 8] is that the correlation between the activations of a pre-trained VGG [33] network has remarkable ability to capture the visual style of an image.”
Universal Style Transfer via Feature Transforms (2017, methodology): “Numerous methods have since been developed to address different aspects including speed [27, 19, 16], quality [28, 18, 32, 31], user control [10], diversity [29, 20], semantics understanding [7, 1] and photorealism [23].”
Language-Driven Image Style Transfer (2021, background): “Artistic Style Transfer Style transfer [1, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56] is to redraw an image with a speciﬁc style.”
Two-Stream Convolutional Networks for Dynamic Texture Synthesis (2017, background): “This has been explored previously for artistic style transfer [4, 15] with static imagery.”
I Lead, You Help but Only with Enough Details: Understanding User Experience of Co-Creation with Artificial Intelligence (2018, background): “also developed a neural network for blending the content of one image and the style of another image [13].”
Guided Image Inpainting: Replacing an Image Region by Pulling Content From Another Image (2018, background): “Style Transfer: Image style transfer combines the highlevel content of one image with the low-level style of another [13, 21, 14, 12].”
Multi-style Generative Network for Real-time Transfer (2017, methodology): “Controlling brush stroke size can be achieved using optimization-based approach [16].”
Recovering Realistic Texture in Image Super-Resolution by Deep Spatial Feature Transform (2018, methodology): “[14] use semantic maps to control perceptual factors in neural style transfer.”
More Control for Free! Image Synthesis with Semantic Diffusion Guidance (2021, background): “Many style transfer works fall under this category where the content of the input image must be preserved while the style of the reference image is transferred [13,29,16,14,26,20,31,30,38,1], yet they struggle to generate diverse images.”
Image2StyleGAN++: How to Edit the Embedded Images? (2019, background): “Local style transfer[13] modifies a region in the input image x to transform it to the style defined by a style reference image.”
TextureGAN: Controlling Deep Image Synthesis with Texture Patches (2017, methodology): “Shortly after, there have been many work on improving [8, 9] from the aspects for generalization [15, 22, 39], efficiency [17, 33] and controllability [10].”
Shift-Net: Image Inpainting via Deep Feature Rearrangement (2018, background): “In the recent few years, style transfer [4, 8, 10, 11, 13, 19, 27, 29, 40] has been an active research topic.”
Neural Style Transfer: A Review (2017, methodology): “’s algorithm in [60], which is a coarse-to-fine stylisation procedure [62].”
The Unreasonable Effectiveness of Texture Transfer for Single Image Super-resolution (2018, methodology): “Gatys et al. [40] demonstrated the spatial control of texture transfer using guided Gram matrices where binary masks are used as guidance channels in order to constrain the textures.”
Inversion-based Style Transfer with Diffusion Models (2022, methodology): “In recent years pre-trained deep convolutional neural networks are used to extract the statistical distribution of features which can capture style patterns effectively [17, 18, 25].”
Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization (2017, methodology): “This is achieved by performing AdaIN separately to different regions in the content feature maps using statistics from different style inputs, similar to [4, 17] but in a completely feed-forward manner.”