The Contextual Loss for Image Transformation with Non-Aligned Data

By Roey Mechrez, Itamar Talmi, Lihi Zelnik-Manor. European Conference on Computer Vision, 2018

421 citations recorded by the source corpus.

Feed-forward CNNs trained for image transformation problems rely on loss functions that measure the similarity between the generated image and a target image. Most of the common loss functions assume that these images are spatially aligned and compare pixels at corresponding locations. However, for many tasks, aligned training pairs of images will not be available. We present an alternative loss function that does not require alignment, thus providing an effective and simple solution for a new space of problems. Our loss is based on both context and semantics -- it compares regions with similar semantic meaning, while considering the context of the entire image. Hence, for example, when transferring the style of one face to another, it will translate eyes-to-eyes and mouth-to-mouth. Our code can be found at this https URL

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/d6e8e560100c43ea22070535b2d0e31f518ffb72

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Deep ViT Features as Dense Visual Descriptors (2021, background): “These representations were shown to align well with human perception [16, 23, 54, 33] and to encode a wide range of visual information - from low level features (e.”
The 2018 PIRM Challenge on Perceptual Image Super-Resolution (2018, background): “An alternative for the perceptual loss used by [28] is the contextual loss [24, 25], that encourages the reconstructed images to have the same statistics as of the high resolution ground-truth images.”
Deep Learning for Single Image Super-Resolution: A Brief Review (2018, methodology): “Based on nonparametric estimation, recently, the contextual loss [115], [116] was proposed by Mechrez et al.”
Language-Driven Image Style Transfer (2021, background): “Artistic Style Transfer Style transfer [1, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56] is to redraw an image with a speciﬁc style.”
Alleviating Semantics Distortion in Unsupervised Low-Level Image-to-Image Translation via Structure Consistency Constraint (2022, methodology): “, CycleGAN, CUT [43], Contexual loss [39], U-GATIT [25], MUNIT [17], may change the geometry structure of input images and potentially cause the semantics mismatch between input and translated images.”
Rethinking Lossy Compression: The Rate-Distortion-Perception Tradeoff (2019, background): “9), and is commonly used in combination with an additional distortion term, e.g. (cid:96) 2 /(cid:96) 1 / contextual loss (Ledig et al., 2017; Mechrez et al., 2018a;b; Liu et al., 2018; Wang et al., 2018; Shama et al., 2018; Shoshan et al., 2018).”
StyleFormer: Real-time Arbitrary Style Transfer via Parametric Style Composition (2021, background): “The existing attempts either tried to adjust the holistic statistics of the content features with that of the style features [17, 18, 19, 20, 21, 22, 23], or non-locally swap the relevant style features in order to match the content features [24, 25, 26, 27, 28, 29, 30, 31].”
SinGAN: Learning a Generative Model From a Single Natural Image (2019, methodology): “Our method outperforms style transfer methods [38, 17] in terms of visual quality (Fig.”
Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior (2023, methodology): “We evaluate these two aspects using the following metrics: 1) LPIPS [61], which assesses the reconstruction quality at the reference view, 2) contextual distance [21], which measures pixel-level similarity between novel-view rendering and the reference, and 3) CLIP score [33], which evaluates the se”
Real-World Single Image Super-Resolution: A Brief Review (2021, methodology): “Inspired by the Contextual Loss [142] and the edge-preserving bilateral ﬁlter [143], Zhang et al. [83] propose the Contextual Bilateral loss (CoBi) to resolve this issue.”
Neural Style Transfer: A Review (2017, background): “[67] propose an alternative contextual loss to realise semantic style transfer in a segmentation-free manner.”
Analyzing Perception-Distortion Tradeoff using Enhanced Perceptual Super-resolution Network (2018, background): “Dataset Scores ENet[39] CX[34] To perform a region-wise comparison, we train both BNet and EPSR with a diﬀerent set of weights for MSE loss and adversarial loss.”
Contrastive Learning for Unpaired Image-to-Image Translation (2020, methodology): “To measure the similarity between two distributions, the Contextual Loss [49] used softmax over cosine disntances of features extracted from pre-trained networks.”
Wide-Context Semantic Image Extrapolation (2019, methodology): “The IDMRF loss [29, 44] between Ŷ Ω and Y L is defined as”
Learning to Maintain Natural Image Statistics (2018, methodology): “Training with a statistical loss: To train a generator network G we compare the output image G(s)≡x with the corresponding target image y via a statistical loss — the Contextual loss [23] — that compares their feature distributions.”
Style Transfer by Relaxed Optimal Transport and Self-Similarity (2019, methodology): “Content Style Ours Reshuffle [6] Gatys [4] CNNMRF [15] Contextual [20]”
NeRF++: Analyzing and Improving Neural Radiance Fields (2020, background): “Robust loss functions, such as the contextual loss (Mechrez et al., 2018), could be applied.”
Cross-Domain Correspondence Learning for Exemplar-Based Image Translation (2020, methodology): “To this end, we employ the contextual loss proposed in [35] to match the statistics be-”