Visual attribute transfer through deep image analogy

By Jing Liao, Yuan Yao, Lu Yuan, Gang Hua, Sing Bing Kang. 2017

548 citations recorded by the source corpus.

We propose a new technique for visual attribute transfer across images that may have very different appearance but have perceptually similar semantic structure. By visual attribute transfer, we mean transfer of visual information (such as color, tone, texture, and style) from one image to another. For example, one image could be that of a painting or a sketch while the other is a photo of a real scene, and both depict the same type of scene. Our technique finds semantically-meaningful dense correspondences between two input images. To accomplish this, it adapts the notion of "image analogy" [Hertzmann et al. 2001] with features extracted from a Deep Convolutional Neutral Network for matching; we call our technique deep image analogy. A coarse-to-fine strategy is used to compute the nearest-neighbor field for generating the results. We validate the effectiveness of our proposed method in a variety of cases, including style/texture transfer, color/style swap, sketch/painting to photo, and time lapse.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Rewriting a Deep Generative Model (2020, methodology): “…have become widely-used tools in the community, enabling applications such as manipulating the semantics of an input scene [60,6,72,16], image colorization [83,34,52,85], photo stylization [25,38,56,54], image-to-image translation [35,86,9,73,55,32], and face editing and synthesis [24,58,63].”
Universal Style Transfer via Feature Transforms (2017, methodology): “Existing style transfer methods are mostly example-based [13, 25, 24, 7, 21].”
DA-GAN: Instance-Level Image Translation by Deep Attention Generative Adversarial Networks (2018, methodology): “We also compare our results with the ones produces by VAT [19] in Figure 8(a).”
CartoonGAN: Generative Adversarial Networks for Photo Cartoonization (2018, methodology): “Liao et al. [21] proposed a Deep Analogy method which keeps semantically meaningful dense correspondences between the content and style images while transferring the style.”
Style Transfer by Relaxed Optimal Transport and Self-Similarity (2019, methodology): “Distinct from the framework of Neural Style Transfer, there are several recent methods [17, 1] that use similarities between deep neural features to build a correspondence map between the content image and style image, and warp the style image onto the content image.”
Neural best-buddies (2018, methodology): “Deep Image Analogy is designed for visual attribute transfer (combining the content of one image with the style of another).”
Photorealistic Style Transfer via Wavelet Transforms (2019, background): “[8], many artistic style transfer studies have been proposed to synthesize stylized images through either iterative optimization [9], finding dense correspondence [20, 28, 10] or manipulating features in pre-trained networks [13, 18].”
Spatially Controllable Image Synthesis with Internal Representation Collaging (2018, background): “Numerous approaches have taken advantage of this property of CNN to transfer the styles/attributes of one image to another [9, 38, 23].”
Deep Sketch-Guided Cartoon Video Inbetweening (2021, methodology): “[29] perform PatchMatch [30] in a deep feature pyramid to”
Cross-Domain Correspondence Learning for Exemplar-Based Image Translation (2020, background): “Following works further improve the correspondence quality by incorporating additional annotations [51, 7, 11, 12, 21, 25], adopting coarse-to-fine strategy [27] or retaining reliable sparse matchings [1].”
StyleFormer: Real-time Arbitrary Style Transfer via Parametric Style Composition (2021, background): “The existing attempts either tried to adjust the holistic statistics of the content features with that of the style features [17, 18, 19, 20, 21, 22, 23], or non-locally swap the relevant style features in order to match the content features [24, 25, 26, 27, 28, 29, 30, 31].”
Enhancing Photorealism Enhancement (2021, background): “retrieved nearest neighbor patches in a learned feature space [46].”
Avatar-Net: Multi-scale Zero-Shot Style Transfer by Feature Decoration (2018, methodology): “Although these methods produce appealing stylized images [26, 21], dense correspondences are limited to a pair of images with similar contents, thus inapplicable to zeroshot style transfer.”
Unpaired Sentiment-to-Sentiment Translation: A Cycled Reinforcement Learning Approach (2018, background): “Style transfer in computer vision has been studied (Johnson et al., 2016; Gatys et al., 2016; Liao et al., 2017; Li et al., 2017; Zhu et al., 2017).”
Improving Shape Deformation in Unsupervised Image-to-Image Translation (2018, methodology): “Recent research has extended the model to perform visual attribute transfer using neural networks [25,13].”
A Style-Aware Content Loss for Real-time HD Style Transfer (2018, methodology): “Inspired by Image Analogies [18], approaches [11,30,39,40] are based on finding dense correspondence between content and style image and often require image pairs to depict similar content.”
Multi-content GAN for Few-Shot Font Style Transfer (2017, unknown): “CNN features are successfully used to represent image styles, and serve as the basis for optimization [7, 16, 20].”