Universal Style Transfer via Feature Transforms

By Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, Ming-Hsuan Yang. 2017

1,142 citations recorded by the source corpus.

Universal style transfer aims to transfer arbitrary visual styles to content images. Existing feed-forward based methods, while enjoying the inference efficiency, are mainly limited by inability of generalizing to unseen styles or compromised visual quality. In this paper, we present a simple yet effective method that tackles these limitations without training on any pre-defined styles. The key ingredient of our method is a pair of feature transforms, whitening and coloring, that are embedded to an image reconstruction network. The whitening and coloring transforms reflect a direct matching of feature covariance of the content image to a given style image, which shares similar spirits with the optimization of Gram matrix based cost in neural style transfer. We demonstrate the effectiveness of our algorithm by generating high-quality stylized images with comparisons to a number of recent methods. We also analyze our method by visualizing the whitened features and synthesizing textures via simple feature coloring.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

A Style-Based Generator Architecture for Generative Adversarial Networks (2018, background): “Compared to more general feature transforms [35, 53], AdaIN is particularly well suited for our purposes due to its efficiency and compact representation.”
Multimodal Unsupervised Image-to-Image Translation (2018, methodology): “10, we compare out method with classical style transfer algorithms including Gatys et al. [5], Chen et al. [85], AdaIN [54], and WCT [55].”
Slimmable Neural Networks (2018, methodology): “Conditional normalization is widely used in tasks including style transfer (Dumoulin et al., 2016; Li et al., 2017a; Huang & Belongie, 2017; Li et al., 2017b), image recognition (Li et al., 2016b; Yang et al., 2018) and many others (Perez et al., 2017b;a).”
Arbitrary Style Transfer With Style-Attentional Networks (2018, methodology): “For arbitrary style transfer, a few methods [13, 7, 20] holistically adjust the content features to match the secondorder statistics of the style features.”
DCAN: Dual Channel-wise Alignment Networks for Unsupervised Scene Adaptation (2018, methodology): “We ﬁx the weights of the encoder, following [52,17], to map images x si and x tj into ﬁxed representations: F si = f gen ( x si ) and F tj = f gen ( x tj ), respectively.”
StyTr2: Image Style Transfer with Transformers (2021, methodology): “style transfer networks [2, 3, 22, 29, 30, 33, 38, 52, 54] are optimized by aligning second-order statistics of content images to style images and can generate stylized results in a feed-forward manner efficiently.”
Image Style Transfer using Convolutional Neural Networks (2018, background): “Li et al. [4] train their decoder only on content images, thus their decoder does not need to know anything about styles and their approach is generalizing even better.”
Big but Imperceptible Adversarial Perturbations via Semantic Manipulation (2019, background): “The Convolutional Neural Network (CNN) based texture transfer from Gatys et al. [11] led to a series of new ideas in the domain of artistic style transfer [12, 18, 23, 32].”
StyleFormer: Real-time Arbitrary Style Transfer via Parametric Style Composition (2021, methodology): “When compared with the prior optimization-based [1, 3] and feed-forward based approaches [18, 17, 25, 19, 23, 35], our method achieves the SOTA stylization re-sults in terms of both visual quality and efficiency.”
Infrared and Visible Image Fusion with ResNet and zero-phase component analysis (2018, methodology): “Li et al.[24] proposed a universal style transfer algorithm using ZCA operation to transfer the style of artistic image into content image.”
A Style-Aware Content Loss for Real-time HD Style Transfer (2018, methodology): “[13], who have relied on powerful but slow iterative optimization, there has recently been a focus on feedforward generator networks [24,47,42,44,29,7,21].”
Photorealistic Style Transfer via Wavelet Transforms (2019, methodology): “Following WCT [18], we report the covariance matrix difference (VGG style loss [8]) between the style image and the outputs of each model.”
Enhancing Photorealism Enhancement (2021, methodology): “Subsequent work introduced increasingly sophisticated techniques for photographic style transfer between images [18], [48], [49], [50], [51], [52], [53], [54].”
Few-shot Image Generation with Elastic Weight Consolidation (2020, methodology): “Example-based schemes work with only one style example but have limitations, such as requiring alignment [9] or only transferring the color and texture [6, 12, 20, 21].”
Dynamic Instance Normalization for Arbitrary Style Transfer (2019, methodology): “In particular, we would like to introduce DIN to the context of domain adaption (Dundar et al. 2018), since style transfer is intrinsically a domain-adaption task (Li et al. 2017a; Li et al. 2019).”
Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, methodology): “Style transfer is an important image editing task that enables the creation of new artistic works [240], [241].”