E-LPIPS: Robust Perceptual Image Similarity via Random Transformation Ensembles

By Markus Kettunen, Erik Härkönen, Jaakko Lehtinen. 2019

82 citations recorded by the source corpus.

It has been recently shown that the hidden variables of convolutional neural networks make for an efficient perceptual similarity metric that accurately predicts human judgment on relative image similarity assessment. First, we show that such learned perceptual similarity metrics (LPIPS) are susceptible to adversarial attacks that dramatically contradict human visual similarity judgment. While this is not surprising in light of neural networks' well-known weakness to adversarial perturbations, we proceed to show that self-ensembling with an infinite family of random transformations of the input --- a technique known not to render classification networks robust --- is enough to turn the metric robust against attack, while retaining predictive power on human judgments. Finally, we study the geometry imposed by our our novel self-ensembled metric (E-LPIPS) on the space of natural images. We find evidence of "perceptual convexity" by showing that convex combinations of similar-looking images retain appearance, and that discrete geodesics yield meaningful frame interpolation and texture morphing, all without explicit correspondences.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Training Generative Adversarial Networks with Limited Data (2020, background): “In an interesting parallel, it has been shown that loss functions can be made significantly more robust in regression settings by using similar image augmentation ensembles [23].”
Stateful Detection of Black-Box Adversarial Attacks (2019, background): “Recent work on robust similarity [25] could also be useful for improving our scheme.”
DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data (2023, background): “Further improvements include ensembling for robustness [45], antialiasing for stability [24, 36, 93], and global descriptors for texture [24].”
Smooth Adversarial Training (2020, methodology): “Besides training models on adversarial data, other ways for improving adversarial robustness include defensive distillation [47], gradient discretization [6, 52, 72], dynamic network architectures [12, 66, 67, 34, 31, 37], randomized transformations [73, 4, 71, 50, 29, 1],”
Deep Image Deblurring: A Survey (2022, methodology): “(2017) dataset, and the datasets from Köhler et al. (2012) and Shen et al.”
Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields (2021, methodology): “See the supplement for renderings from SVS that achieve lower LPIPS scores than our model despite having reduced image quality [20].”