A Closed-form Solution to Photorealistic Image Stylization

By Yijun Li, Ming-Yu Liu, Xueting Li, Ming-Hsuan Yang, Jan Kautz. European Conference on Computer Vision, 2018

447 citations recorded by the source corpus.

Photorealistic image stylization concerns transferring style of a reference photo to a content photo with the constraint that the stylized photo should remain photorealistic. While several photorealistic image stylization methods exist, they tend to generate spatially inconsistent stylizations with noticeable artifacts. In this paper, we propose a method to address these issues. The proposed method consists of a stylization step and a smoothing step. While the stylization step transfers the style of the reference photo to the content photo, the smoothing step ensures spatially consistent stylizations. Each of the steps has a closed-form solution and can be computed efficiently. We conduct extensive experimental validations. The results show that the proposed method generates photorealistic stylization outputs that are more preferred by human subjects as compared to those by the competing methods while running much faster. Source code and additional results are available at this https URL .

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/eadac91600c67ee2343e7ce76934e92b1796e60e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Rethinking and Improving the Robustness of Image Style Transfer (2021, background): “A consistent observation from this work [43, 24, 30, 23, 25, 19, 9, 2, 8] is that the correlation between the activations of a pre-trained VGG [33] network has remarkable ability to capture the visual style of an image.”
StyTr2: Image Style Transfer with Transformers (2021, unknown): “style transfer networks [2, 3, 22, 29, 30, 33, 38, 52, 54] are optimized by aligning second-order statistics of content images to style images and can generate stylized results in a feed-forward manner efficiently.”
Style Injection in Diffusion: A Training-Free Approach for Adapting Large-Scale Diffusion Models for Style Transfer (2023, unknown): “Neural style transfer [11, 24–26, 29, 31, 43, 44, 52] is an example-guided image generation task that transfers the style of one image onto another while retaining the content of the original.”
Exemplar Guided Unsupervised Image-to-Image Translation (2018, methodology): “To avoid this issue, previous style transfer works [8, 25, 28] take a semantic label map as additional input.”
R2D2: Repeatable and Reliable Detector and Descriptor (2019, background): “3, in equal proportions: random web images (W), Aachen-day images (A), Aachen-night images obtained from automatic style transfer [24]3 (S).”
Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learning (2020, unknown): “[27].”
Style Transfer by Relaxed Optimal Transport and Self-Similarity (2019, background): “The most common quantitative evaluation is asking users to rank the output of each algorithm according to aesthetic appeal [6, 16, 19].”
RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whitening (2021, background): “Whitening all covariance elements may diminish feature discrimination [45, 61] and distort the boundary of an object [31, 30] because domain-speciﬁc style and domain-invariant content are simultaneously encoded in the covariance of the feature map.”
Semantic photo manipulation with a generative image prior (2019, background): “Recent examples include image inpainting [Iizuka et al. 2017; Pathak et al. 2016; Yu et al. 2018], image colorization [Iizuka et al. 2016; Zhang et al. 2016b], photo stylization [Gatys et al. 2016; Li et al. 2018; Zhu et al. 2017] and photo enhancement [Gharbi et al. 2017; Kim and Park 2018].”
Few-shot Image Generation with Elastic Weight Consolidation (2020, methodology): “Example-based schemes work with only one style example but have limitations, such as requiring alignment [9] or only transferring the color and texture [6, 12, 20, 21].”
Rethinking Style Transfer: From Pixels to Parameterized Brushstrokes (2021, background): “There are papers on feed-forward architectures [23, 48], universal feed-forward models [18, 32, 33, 31], disentanglement of style and content [44, 28, 29], ultra-resolution models [50], meta-learning techniques [45, 56], and video style transfer [4].”
Deep Single-Image Portrait Relighting (2019, methodology): “[16] is designed for general photo style transfer, similarly we use the L channel for portrait relighting.”
Enhancing Photorealism Enhancement (2021, methodology): “Subsequent work introduced increasingly sophisticated techniques for photographic style transfer between images [18], [48], [49], [50], [51], [52], [53], [54].”
Swapping Autoencoder for Deep Image Manipulation (2020, background): “Photorealistic style transfer methods further constrain the result to be represented by local affine color transforms from the input image [54, 49, 78], but such methods only allow local color changes.”
Language-Driven Image Style Transfer (2021, background): “Photorealistic style transfer [57, 58, 59, 60, 61, 62] aims at applying reference styles on scenes without hurting details and satisfying contradictory objectives.”
Photorealistic Style Transfer via Wavelet Transforms (2019, methodology): “[19] proposed a photorealistic variant of WCT (PhotoWCT).”
Learning Texture Invariant Representation for Domain Adaptation of Semantic Segmentation (2020, unknown): “The photo-realistic style transfer algorithm [15] is another option, which preserves the precise structure of the original image using a smoothing step after the stylization step.”
Manipulating Attributes of Natural Scenes via Hallucination (2018, methodology): “Here, we only provide results obtained by using FPST [Li et al. 2018], as it gives slightly better results in our experiments and also significantly faster than DPST [Luan et al.”