A Style-Aware Content Loss for Real-time HD Style Transfer

By Artsiom Sanakoyeu, Dmytro Kotovenko, Sabine Lang, Björn Ommer. 2018

230 citations recorded by the source corpus.

Recently, style transfer has received a lot of attention. While much of this research has aimed at speeding up processing, the approaches are still lacking from a principled, art historical standpoint: a style is more than just a single image or an artist, but previous work is limited to only a single instance of a style or shows no benefit from more images. Moreover, previous work has relied on a direct comparison of art in the domain of RGB images or on CNNs pre-trained on ImageNet, which requires millions of labeled object bounding boxes and can introduce an extra bias, since it has been assembled without artistic consideration. To circumvent these issues, we propose a style-aware content loss, which is trained jointly with a deep encoder-decoder network for real-time, high-resolution stylization of images and videos. We propose a quantitative measure for evaluating the quality of a stylized image and also have art historians rank patches from our approach against those from previous work. These and our qualitative results ranging from small image patches to megapixel stylistic images and videos show that our approach better captures the subtle nature in which a style affects content.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

AutoGAN-Distiller: Searching to Compress Generative Adversarial Networks (2020, neutral): “These encoder-decoder GANs also gain popularity in image enhancement (Wang et al., 2018a; Kupyn et al., 2019; Jiang et al., 2019) and editing (Sanakoyeu et al., 2018; Yang et al., 2019).”
Controllable Artistic Text Style Transfer via Shape-Matching GAN (2019, background): “Meanwhile, other researchers regard style transfer as an image-to-image translation problem [16, 32], and exploited Generative Adversarial Network (GAN) [12] to transfer specialized styles such as cartoons [7], paintings [25] and makeups [8, 5].”
StyleFormer: Real-time Arbitrary Style Transfer via Parametric Style Composition (2021, background): “The second line of works [44, 45, 46, 47] exploited the power of generative adversarial network to generate the stylized images.”
Rethinking Style Transfer: From Pixels to Parameterized Brushstrokes (2021, background): “There are papers on feed-forward architectures [23, 48], universal feed-forward models [18, 32, 33, 31], disentanglement of style and content [44, 28, 29], ultra-resolution models [50], meta-learning techniques [45, 56], and video style transfer [4].”
Language-Driven Image Style Transfer (2021, background): “By contrast, artistic style transfer [63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 2, 3, 83, 84, 85, 86] captures style concepts from reference and modiﬁes color distributions and texture patterns of content images.”
Style Transfer by Relaxed Optimal Transport and Self-Similarity (2019, methodology): “[24] propose two new forms of quantitative evaluation.”