Demystifying Neural Style Transfer

By Yanghao Li, Naiyan Wang, Jiaying Liu, Xiaodi Hou. 2017

566 citations recorded by the source corpus.

Neural Style Transfer has recently demonstrated very exciting results which catches eyes in both academia and industry. Despite the amazing results, the principle of neural style transfer, especially why the Gram matrices could represent style remains unclear. In this paper, we propose a novel interpretation of neural style transfer by treating it as a domain adaptation problem. Specifically, we theoretically show that matching the Gram matrices of feature maps is equivalent to minimize the Maximum Mean Discrepancy (MMD) with the second order polynomial kernel. Thus, we argue that the essence of neural style transfer is to match the feature distributions between the style images and the generated images. To further support our standpoint, we experiment with several other distribution alignment methods, and achieve appealing results. We believe this novel interpretation connects these two important research fields, and could enlighten future researches.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

A Style-Based Generator Architecture for Generative Adversarial Networks (2018, background): “) reliably encode the style of an image [19, 36] while spatially varying features encode a specific instance.”
Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization (2017, background): “[33] recently showed that matching many other statistics, including channel-wise mean and variance, are also effective for style transfer.”
Slimmable Neural Networks (2018, methodology): “Conditional normalization is widely used in tasks including style transfer (Dumoulin et al., 2016; Li et al., 2017a; Huang & Belongie, 2017; Li et al., 2017b), image recognition (Li et al., 2016b; Yang et al., 2018) and many others (Perez et al., 2017b;a).”
Style Transfer in Text: Exploration and Evaluation (2017, background): “Li et al. (2017) proposes to treat style transfer as a domain adaptation problem.”
Convolutional Networks with Adaptive Inference Graphs (2017, background): “Since operating on the full feature map is computationally expensive, we build upon recent studies (Hu et al. 2017; Huang and Belongie 2017; Li et al. 2017) which show that much of the information in convolutional features is captured by the statistics of the different channels and their interdepend”
Editing in Style: Uncovering the Local Semantics of GANs (2020, background): “This style-based control mechanism is motivated by style transfer [8], [20], where it has been shown that manipulating per-channel mean and variance is sufficient to control the style of an image [14].”
Deep Visual Domain Adaptation: A Survey (2018, background): “[46] demonstrated that matching the Gram matrices of feature maps is equivalent to minimizing the MMD.”
Arbitrary Style Transfer With Style-Attentional Networks (2018, background): “Several approaches [1, 8, 12, 22, 3, 14, 19, 26, 29] have been developed based on feedforward networks.”
SRM: A Style-Based Recalibration Module for Convolutional Neural Networks (2019, background): “[21] also addressed style transfer by matching a variety of CNN feature statistics such as linear, polynomial and Gaussian kernels.”
Image-Image Domain Adaptation with Preserved Self-Similarity and Domain-Dissimilarity for Person Re-identification (2017, background): “Neural style transfer [12, 23, 43, 21, 5, 24, 19, 25] is another strategy of image-image translation, which aims at replicating the style of one image, while our work focuses on learning the mapping function between two domains, rather than two images.”
DCAN: Dual Channel-wise Alignment Networks for Unsupervised Scene Adaptation (2018, unknown): “Though style transfer can be seen as a special domain adaptation problem with each style as a domain [50], our goal in this work is diﬀerent: we focus on unsupervised scene adaption, by jointly synthesizing images and performing segmentation with the help of images from the target domain for…”
Dynamic Instance Normalization for Arbitrary Style Transfer (2019, methodology): “We use the perceptual loss proposed in (Johnson, Alahi, and Fei-Fei 2016) as our content loss, and the BN-statistic loss proposed in (Li et al. 2017a) as our style loss, which are widely adopted in NST.”
A Comprehensive Survey on Test-Time Adaptation Under Distribution Shifts (2023, unknown): “As indicated in [183], matching the BN statistics can aid in ensuring that the generated data resembles the source style.”
Unpaired Sentiment-to-Sentiment Translation: A Cycled Reinforcement Learning Approach (2018, background): “Style transfer in computer vision has been studied (Johnson et al., 2016; Gatys et al., 2016; Liao et al., 2017; Li et al., 2017; Zhu et al., 2017).”
StyTr2: Image Style Transfer with Transformers (2021, methodology): “Optimizationbased style transfer methods [19, 31, 47] render the input content images with learned style representation iteratively.”
Image Style Transfer using Convolutional Neural Networks (2018, background): “It was shown that various different statistics on the style image can be used to capture its style [13].”
StyleFormer: Real-time Arbitrary Style Transfer via Parametric Style Composition (2021, background): “The works [5] and [7] explored how to control the perceptual factors during the transfer process, and another two works [37] and [38] demystified the components proposed in [1].”
Domain Generalization Using a Mixture of Multiple Latent Domains (2019, methodology): “TF (Li et al. 2017a): The low-rank parameterized neural network, which reduces the number of parameters to be trained.”
Real-Time Monocular Depth Estimation Using Synthetic Data with Domain Adaptation via Image Style Transfer (2018, methodology): “As demonstrated in [47], style transfer can be considered as a distribution alignment process from the content image to the style image [34].”
Image Quality Assessment: Unifying Structure and Texture Similarity (2020, background): “Subsequent results suggest that a reduced set of statistics, containing only the mean and variance of CNN channels, is sufficient for texture classification or style transfer [55]–[57].”