Deep Image Harmonization

By Yi-Hsuan Tsai, Xiaohui Shen, Zhe L. Lin, Kalyan Sunkavalli, Xin Lu, Ming-Hsuan Yang. 2017

307 citations recorded by the source corpus.

Compositing is one of the most common operations in photo editing. To generate realistic composites, the appearances of foreground and background need to be adjusted to make them compatible. Previous approaches to harmonize composites have focused on learning statistical relationships between hand-crafted appearance features of the foreground and background, which is unreliable especially when the contents in the two layers are vastly different. In this work, we propose an end-to-end deep convolutional neural network for image harmonization, which can capture both the context and semantic information of the composite images during harmonization. We also introduce an efficient way to collect large-scale and high-quality training data that can facilitate the training process. Experiments on the synthesized dataset and real composite images show that the proposed network outperforms previous state-of-the-art methods.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

GAN Inversion: A Survey (2021, methodology): “In recent years, GANs have been applied to computer vision tasks ranging from image translation [2], [3], [4], image manipulation [5], [6], [7], to image restoration [8], [9], [10].”
Learning to Adapt Structured Output Space for Semantic Segmentation (2018, methodology): “Recently, methods based on convolutional neural networks (CNNs) have achieved significant progress in semantic segmentation [2, 21, 23, 24, 38, 40, 41] with applications for autonomous driving [9] and image editing [35].”
GP-GAN: Towards Realistic High-Resolution Image Blending (2017, methodology): “Our work is different from these gradient-based approaches in that we introduce GANs to generate a low-resolution realistic image as the color constraint, resulting in a more natural composite image. [32, 35, 39] also address a similar task to ours. However, they focus on adjusting the color and ill”
Semi-Parametric Image Synthesis (2018, background): “[31] train a convolutional network to harmonize the appearance of image composites.”
Compositional GAN: Learning Image-Conditional Binary Composition (2018, background): “Tsai et al. (2017) developed an end-to-end deep CNN for image harmonization to automatically capture the context and semantic information of the composite image.”
Low-Light Image Enhancement via a Deep Hybrid Network (2019, methodology): “Inspired by the effectiveness of encoder-decoder networks in image denoising [45], dehazing [46], [47], inpainting [48], matting [49], and harmonization [50], we construct our content branch based on the residual encoder-decoder architecture with specific designs for content prediction as follows: 1”
Fighting Fake News: Image Splice Detection via Learned Self-Consistency (2018, background): “With only rudimentary editing skills, it is now possible to create realistic image composites [3, 4], fill in large image regions [1, 5, 6], generate plausible video from speech [7, 8], etc.”
DoveNet: Deep Image Harmonization via Domain Verification (2019, methodology): “Considering the unavailability and shortcomings of the dataset built in [43], we tend to build our own stronger dataset.”
Paint by Example: Exemplar-based Image Editing with Diffusion Models (2022, background): “This task is completely different from image harmonization [20,64] that mainly focuses on color and lighting correction when compositing the foreground objects, whereas we aim for a much more complex job: semantically transforming the exemplar, e.”
Guided Image Inpainting: Replacing an Image Region by Pulling Content From Another Image (2018, methodology): “Image Harmonization: Image harmonization methods adjust low-level appearances of foreground and background regions to make them compatible when generating realistic composite images [34, 35, 30, 31, 29, 36].”
Gated Fusion Network for Single Image Dehazing (2018, methodology): “We use an encoder-decoder network, which has been shown to produce good results for a number of generative tasks such as image denoising [20], image harmonization [37], time-lapse video generation [38].”
Compositional GAN: Learning Conditional Image Composition (2018, unknown): “[31] developed an end-to-end deep CNN for image harmonization which could automatically capture the context and semantic information of the composite image and was able to outperform its precedents [30, 33] which used transferring statistics of hand-crafted features including color and texture to…”
FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape From Single RGB Images (2019, background): “[31] and the deep image colorization approach by Zhang et al .”
Image Inpainting using Block-wise Procedural Training with Annealed Adversarial Counterpart (2018, methodology): “To train this model, the data acquisition is similar to [23], where we create artiﬁcially composited images using statistics color transfer [21] and another randomly chosen guide image.”
Knowledge As Priors: Cross-Modal Knowledge Generalization for Datasets Without Superior Knowledge (2020, methodology): “…the original images collected with the green screen background ( G ) in FreiHAND, together with their post-processed results using three different strategies: harmonization [46] ( H ), colorization auto [57] ( A ), colorization sample [57] ( S ), as three different domains contained by FreiHAND.”