Deep Photo Style Transfer

By Fujun Luan, Sylvain Paris, Eli Shechtman, Kavita Bala. 2017

757 citations recorded by the source corpus.

This paper introduces a deep-learning approach to photographic style transfer that handles a large variety of image content while faithfully transferring the reference style. Our approach builds upon the recent work on painterly transfer that separates style from the content of an image by considering different layers of a neural network. However, as is, this approach is not suitable for photorealistic style transfer. Even when both the input and reference images are photographs, the output still exhibits distortions reminiscent of a painting. Our contribution is to constrain the transformation from the input to the output to be locally affine in colorspace, and to express this constraint as a custom fully differentiable energy term. We show that this approach successfully suppresses distortion and yields satisfying photorealistic style transfers in a broad variety of scenarios, including transfer of the time of day, weather, season, and artistic edits.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Rewriting a Deep Generative Model (2020, background): “Recently, thanks to rapid advances of deep generative models [26,45,31], learning-based image synthesis and editing methods have become widely-used tools in the community, enabling applications such as manipulating the semantics of an input scene [60,6,72,16], image colorization [83,34,52,85], photo”
Greedy AutoAugment (2019, background): “A third example is to help generators and discriminators for training generative adversarial networks [10], [11].”
Latent Space Physics: Towards Learning the Temporal Evolution of Fluid Flow (2018, background): “2016], and to robustly transfer image styles between photographs [Luan et al. 2017], to name just a few examples.”
Lifelong GAN: Continual Learning for Conditional Image Generation (2019, background): “Moreover, images can be stylized by disentangling the style and the content [13, 21] or by encoding styles into a stylebank (set of convolution filters) [7].”
Universal Style Transfer via Feature Transforms (2017, methodology): “Numerous methods have since been developed to address different aspects including speed [27, 19, 16], quality [28, 18, 32, 31], user control [10], diversity [29, 20], semantics understanding [7, 1] and photorealism [23].”
Data augmentation for improving deep learning in image classification problem (2018, background): “In some recent works authors proposed the use of this method to change pictures from day to night [7], clear day to rain [25], or other landscape changes [26], however currently it is still applied in very limited fields in contrary to the wide existing possibilities.”
Neural Style Transfer: A Review (2017, methodology): “[83] address the efficiency issue of [81] by handling this problem with two steps, the stylisation step and smoothing step.”
Swapping Autoencoder for Deep Image Manipulation (2020, background): “Photorealistic style transfer methods further constrain the result to be represented by local affine color transforms from the input image [54, 49, 78], but such methods only allow local color changes.”
Language-Driven Image Style Transfer (2021, background): “Photorealistic style transfer [57, 58, 59, 60, 61, 62] aims at applying reference styles on scenes without hurting details and satisfying contradictory objectives.”
Enjoy Your Editing: Controllable GANs for Image Editing via Latent Space Navigation (2021, positive): “Encouraged by the success of deep nets, recent works have applied deep learning methods for semantic image editing tasks such as style transfer (Li et al., 2017; Luan et al., 2017), image-to-image translation (Choi et al.”
Learning Image-Adaptive 3D Lookup Tables for High Performance Photo Enhancement in Real-Time (2020, methodology): “Recently, deep neural networks have been employed to extract more representative features and achieve more powerful and ﬂexible (sometimes artistic) style transfer [31], [32], [33].”
Avatar-Net: Multi-scale Zero-Shot Style Transfer by Feature Decoration (2018, background): “A number of variants have been developed thereafter to adapt this framework to different scenarios and requirements, including photorealistic rendering [23], semantically composite transfer [2], temporal coherence [24] and so on.”
Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Language (2018, methodology): “With the advance of deep learning and generative models, many techniques have been developed for image editing including the style transfer [2, 4, 5], the drawing based editing [6, 7], and the domain/attribute translation [8–10].”
Exemplar Guided Unsupervised Image-to-Image Translation (2018, methodology): “However, due to the lack of semantic consistency, these stylization methods usually generate non-photo-realistic images with the “spills over” problem [28].”
Contrastive Learning for Unpaired Image-to-Image Translation (2020, background): “one image has not been explored by previous image-to-image translation methods. Our painting !photo task is also dierent from neural style transfer [18,33] (photo !painting) and photo style transfer [47,81] (photo !photo). Since the whole image (at HD resolution) cannot t on a commercial GPU, at eac”
Manipulating Attributes of Natural Scenes via Hallucination (2018, methodology): “Hence, in the figure, we both present the reference images generated by our approach and retrieved by the [Luan et al. 2017] respectively between retrieved images and input images, c) : results of our method using deep style transfer method [Luan et al. 2017] between generated image by proposed SGN…”
Structure-Preserving Super Resolution With Gradient Guidance (2020, methodology): “Gradient-Relevant Methods: Gradient information has been utilized in previous work [2, 29].”
The Unreasonable Effectiveness of Texture Transfer for Single Image Super-resolution (2018, methodology): “Afterwards, much work has been done in order to improve the speed [3, 32] and quality [33–35] of style transfer using feed forward networks and perceptual losses.”
A Style-Aware Content Loss for Real-time HD Style Transfer (2018, background): “Beyond that, a number of methods have been proposed to address different aspects of style transfer, including quality [14,49,4,47,23], diversity [28,44], photorealism [32], combining several styles in a single model [45,7,3] and generalizing to previously unseen styles [21,29,15,38].”