Learning a Discriminative Model for the Perception of Realism in Composite Images

By Jun-Yan Zhu, Philipp Krahenbuhl, Eli Shechtman, Alexei A. Efros. 2015

147 citations recorded by the source corpus.

What makes an image appear realistic? In this work, we are answering this question from a data-driven perspective by learning the perception of visual realism directly from large amounts of data. In particular, we train a Convolutional Neural Network (CNN) model that distinguishes natural photographs from automatically generated composite images. The model learns to predict visual realism of a scene in terms of color, lighting and texture compatibility, without any human annotations pertaining to it. Our model outperforms previous works that rely on hand-crafted heuristics, for the task of classifying realistic vs. unrealistic photos. Furthermore, we apply our learned model to compute optimal parameters of a compositing method, to maximize the visual realism score predicted by our CNN model. We demonstrate its advantage against existing methods via a human perception study.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

CNN-Generated Images Are Surprisingly Easy to Spot… for Now (2019, methodology): “Taking the similar approach as previous image realism works [25, 53], we rank synthesized images from each dataset by the model’s prediction, and visualize images in the 0 th , 25 th , 50 th , 75 th , 100 th percentile of the “fakeness” score from our model’s output.”
Generative Visual Manipulation on the Natural Image Manifold (2016, background): “[18] recently showed that discriminative deep neural networks learn a much stronger prior that captures both low-level statistics, as well as higher order semantic or color-balance clues.”
Real-time user-guided image colorization with learned deep priors (2017, background): “s, such as photo enhancement (Yan et al. 2016), sketch simplification (Simo-Serra et al. 2016), style transfer (Gatys et al. 2016; Selim et al. 2016), inpainting (Pathak et al. 2016), image blending (Zhu et al. 2015) and denoising (Gharbi et al. 2016). Most of these works built image filtering pipel”
GP-GAN: Towards Realistic High-Resolution Image Blending (2017, neutral): “ween realistic and artiﬁcial images is incredibly hard. There are no suitable metrics for judging how realistic and natural an image is until recently a convolutional neural network called RealismCNN [42] is proposed. RealismCNN learns to predict visual realism of a scene regarding colour, lighting ”
Semi-Parametric Image Synthesis (2018, background): “[40] train a convolutional network to predict the realism of image composites.”
Compositional GAN: Learning Image-Conditional Binary Composition (2018, background): “Zhu et al. (2015) addressed the composition problem by training a discriminator that could distinguish realistic composite images from synthetic ones.”
Deep Image Harmonization (2017, methodology): “Ground truth Input Lalonde [13] Xue [28] Zhu [32] Ours”
ForensicTransfer: Weakly-supervised Domain Adaptation for Forgery Detection (2018, background): “It is now possible to transfer semantic content from a source domain to a target domain characterized by a different style [45], create automatically image compositions [46], and reconstruct missing image parts by semantic inpainting [26, 21].”
Fighting Fake News: Image Splice Detection via Learned Self-Consistency (2018, background): “With only rudimentary editing skills, it is now possible to create realistic image composites [3, 4], fill in large image regions [1, 5, 6], generate plausible video from speech [7, 8], etc.”
Deep Identity-aware Transfer of Facial Attributes (2016, background): “Deep convolutional networks (CNNs) have shown great success in versatile high level vision problems [22, 30, 11, 2], and also exhibited their remarkable power in understanding and generating images at a ﬁne level of granularity [42].”
GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields (2020, background): “To this end, many works [9, 25, 39, 43, 44, 48, 54, 71, 74, 97, 98] investigate how disentangled representations can be learned from data without explicit supervision.”
Compositional GAN: Learning Conditional Image Composition (2018, methodology): “[39] proposed to learn optimal parameters of a compositing method as well as a discriminator (a CNN) that could distinguish realistic composite images from non-realistic ones.”