Maintaining Natural Image Statistics with the Contextual Loss

By Roey Mechrez, Itamar Talmi, Firas Shama, Lihi Zelnik-Manor. 2018

104 citations recorded by the source corpus.

Maintaining natural image statistics is a crucial factor in restoration and generation of realistic looking images. When training CNNs, photorealism is usually attempted by adversarial training (GAN), that pushes the output images to lie on the manifold of natural images. GANs are very powerful, but not perfect. They are hard to train and the results still often suffer from artifacts. In this paper we propose a complementary approach, that could be applied with or without GAN, whose goal is to train a feed-forward CNN to maintain natural internal statistics. We look explicitly at the distribution of features in an image and train the network to generate images with natural feature distributions. Our approach reduces by orders of magnitude the number of images required for training and achieves state-of-the-art results on both single-image super-resolution, and high-resolution surface normal estimation.

Subfields: Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

The Perception-Distortion Tradeoff (2017, neutral): “[74].”
ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks (2018, background): “Contextual loss [30] is developed to generate images with natural image statistics by using an objective that focuses on the feature distribution rather than merely comparing the appearance.”
Rethinking Lossy Compression: The Rate-Distortion-Perception Tradeoff (2019, background): “9), and is commonly used in combination with an additional distortion term, e.g. (cid:96) 2 /(cid:96) 1 / contextual loss (Ledig et al., 2017; Mechrez et al., 2018a;b; Liu et al., 2018; Wang et al., 2018; Shama et al., 2018; Shoshan et al., 2018).”
Deblurring via Stochastic Refinement (2021, background): “To alleviate this problem, recent works introduced additional loss terms [17,21,34,44,45] that seek to improve the quality of generated images under metrics that represent human perception more reliably.”
Contrastive Learning for Unpaired Image-to-Image Translation (2020, background): “ch as SSIM [77] and FSIM [83] on human perceptual tests [86]. In particular, the Contextual Loss [51] boosts the perceptual quality of pretrained VGG features, validated by human perceptual judgments [50]. In these cases, the frozen network weights cannot adapt to the data on hand. Furthermore, the ”