Neural Style Transfer: A Review

By Yongcheng Jing, Yezhou Yang, Zunlei Feng, Jingwen Ye, Mingli Song. 2017

870 citations recorded by the source corpus.

The seminal work of Gatys et al. demonstrated the power of Convolutional Neural Networks (CNNs) in creating artistic imagery by separating and recombining image content and style. This process of using CNNs to render a content image in different styles is referred to as Neural Style Transfer (NST). Since then, NST has become a trending topic both in academic literature and industrial applications. It is receiving increasing attention and a variety of approaches are proposed to either improve or extend the original NST algorithm. In this paper, we aim to provide a comprehensive overview of the current progress towards NST. We first propose a taxonomy of current algorithms in the field of NST. Then, we present several evaluation methods and compare different NST algorithms both qualitatively and quantitatively. The review concludes with a discussion of various applications of NST and open problems for future research. A list of papers discussed in this review, corresponding codes, pre-trained models and more comparison results are publicly available at: https://osf.io/f8tu4/.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Style transfer-based image synthesis as an efficient regularization technique in deep learning (2019, positive): “The first proposition of using CNN feature activations to mimic an artistic style of one image while maintaining the content of another was published in 2015 [11], and since then many different modifications to this method emerged such as: universal style transfer, semantic style transfer (ST), inst”
Multi-style Generative Network for Real-time Transfer (2017, background): “[24], and they point out that no generative methods can adjust the brush stroke size in real-time.”
Learning What Not to Segment: A New Perspective on Few-Shot Segmentation (2022, methodology): “Inspired by the style loss that is extensively adopted in the domain of image style transfer [12,13,20], we first calculate the difference of the Gram matrices of the two input images and then utilize the Frobenius norm to obtain the overall indicator for guiding the adjustment process.”
Adversarial Text-to-Image Synthesis: A Review (2021, methodology): “This approach has successfully been adapted to many applications such as high-resolution synthesis of human faces [3], image super-resolution [4], image in-painting [5, 6], data augmentation [7], style transfer [8, 9], image-to-image translation [10, 11], and representation learning [12, 13].”
Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, background): “Style transfer is an important image editing task that enables the creation of new artistic works [240], [241].”
Real-Time Monocular Depth Estimation Using Synthetic Data with Domain Adaptation via Image Style Transfer (2018, background): “As demonstrated in [47], style transfer can be considered as a distribution alignment process from the content image to the style image [34].”
An Introduction to Image Synthesis with Generative Adversarial Nets (2018, background): “This problem is related to style transfer [68] [69], which takes a content image and a style image and output an image with the content of the content image and the style of the style image.”
The Contextual Loss for Image Transformation with Non-Aligned Data (2018, background): “These differ mostly in the choice of architecture and loss function [6,7,8,20,21], a review is given in [22].”
DiffEdit: Diffusion-based semantic image editing with mask guidance (2022, neutral): “The field of image editing encompasses many different tasks, from photo colorization and retouching (Shi et al., 2020), to style transfer (Jing et al., 2019), inserting objects in images (Gafni & Wolf, 2020; Brown et al., 2022), image-to-image translation (Zhu et al., 2017; Sa-haria et al., 2022a),…”
A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2021, background): “It is essentially a weighted sum of the error (loss) between the input content and the output image and the loss between the original style and the applied style [161].”
Rethinking and Improving the Robustness of Image Style Transfer (2021, neutral): “For an extensive review of stylization methods please see [21].”
Language-Driven Image Style Transfer (2021, background): “Artistic Style Transfer Style transfer [1, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56] is to redraw an image with a speciﬁc style.”
A comprehensive survey of recent trends in deep learning for digital images augmentation (2021, background): “The deep learning image data augmentation included three types, the first type was the image data augmentation using GAN, while the second type was the Neural Style Transfer, and the third type was the meta metric learning.”
AdaAttN: Revisit Attention Mechanism in Arbitrary Neural Style Transfer (2021, methodology): “Since the learned model can only work for one specific style, this method and its following works [40, 34, 35, 21, 27, 39, 16, 20] are categorized to Per-Style-Per-Model method [17].”
Dynamic Instance Normalization for Arbitrary Style Transfer (2019, background): “We use the perceptual loss proposed in (Johnson, Alahi, and Fei-Fei 2016) as our content loss, and the BN-statistic loss proposed in (Li et al. 2017a) as our style loss, which are widely adopted in NST. Similar to prior works (Gatys, Ecker, and Bethge 2016; Huang and Belongie 2017; Jing et al. 2018)”
Enhancing Performance of Deep Learning Models with different Data Augmentation Techniques: A Survey (2020, background): “three images are combine together in such a way that the input image is modified so that it looks like the content image, but “painted” in the style of the style image [24]”
The face of art (2019, methodology): “Current neural style transfer methods fit into one of two categories [Jing et al. 2017]: optimization-based methods [Gatys et al. 2015; Li and Wand 2016; Li et al. 2017], and model-based neural methods [Johnson et al. 2016; Ulyanov et al. 2016].”
Class-Based Styling: Real-Time Localized Style Transfer with Semantic Segmentation (2019, background): “[5] conducted a review of neural style transfer methods, and evaluated their different applications.”