Photorealistic Style Transfer via Wavelet Transforms

By Jaejun Yoo, Youngjung Uh, Sanghyuk Chun, Byeongkyu Kang, Jung-Woo Ha. 2019

425 citations recorded by the source corpus.

Recent style transfer models have provided promising artistic results. However, given a photograph as a reference style, existing methods are limited by spatial distortions or unrealistic artifacts, which should not happen in real photographs. We introduce a theoretically sound correction to the network architecture that remarkably enhances photorealism and faithfully transfers the style. The key ingredient of our method is wavelet transforms that naturally fits in deep networks. We propose a wavelet corrected transfer based on whitening and coloring transforms (WCT2) that allows features to preserve their structural information and statistical properties of VGG feature space during stylization. This is the first and the only end-to-end model that can stylize a 1024x1024 resolution image in 4.7 seconds, giving a pleasing and photorealistic quality without any post-processing. Last but not least, our model provides a stable video stylization without temporal constraints. Our code, generated images, pre-trained models and supplementary documents are all available at https://github.com/ClovaAI/WCT2.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Deep Learning in Medical Imaging (2019, methodology): “Following this research, Yoo et al. 80 devised the wavelet pooling layer to enable photorealistic style transfer.”
Focal Frequency Loss for Image Reconstruction and Synthesis (2020, background): “Besides, several studies have incorporated frequency analysis with network compression [16, 66, 68, 6, 17] and feature reduction [35, 61] to accelerate the training and inference of networks.”
CLIPstyler: Image Style Transfer with a Single Text Condition (2021, background): “[27] proposed a wavelet transform based WCT for better preservation of content information for photo-realistic style transfer.”
Enhancing Photorealism Enhancement (2021, methodology): “Subsequent work introduced increasingly sophisticated techniques for photographic style transfer between images [18], [48], [49], [50], [51], [52], [53], [54].”
Language-Driven Image Style Transfer (2021, background): “Photorealistic style transfer [57, 58, 59, 60, 61, 62] aims at applying reference styles on scenes without hurting details and satisfying contradictory objectives.”
EditGAN: High-Precision Semantic Image Editing (2021, background): “(vi) Finally, another line of research targets primarily very high-level image and photo stylization and global appearance modiﬁcations [37, 41, 46, 52–57].”
Swapping Autoencoder for Deep Image Manipulation (2020, methodology): “Our model is rated best for capturing style, and second-best for preserving content, behind WCT(2) [78], a photorealistic style transfer method.”
Contrastive Learning for Unpaired Image-to-Image Translation (2020, methodology): “[19], WCT [82], STROTSS [39], and patch-based CycleGAN [89].”
ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models (2021, background): “The first approach learns to control by providing the desired condition in training procedure and has shown remarkable performance on various tasks, such as segmentation mask conditioned generation [31, 59], style transfer [9, 50], and inpainting [23, 52].”