Image Quality Assessment: Unifying Structure and Texture Similarity

By Keyan Ding, Kede Ma, Shiqi Wang, Eero P. Simoncelli. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020

1,428 citations recorded by the source corpus.

Objective measures of image quality generally operate by comparing pixels of a “degraded” image to those of the original. Relative to human observers, these measures are overly sensitive to resampling of texture regions (e.g., replacing one patch of grass with another). Here, we develop the first full-reference image quality model with explicit tolerance to texture resampling. Using a convolutional neural network, we construct an injective and differentiable function that transforms images to multi-scale overcomplete representations. We demonstrate empirically that the spatial averages of the feature maps in this representation capture texture appearance, in that they provide a set of sufficient statistical constraints to synthesize a wide variety of texture patterns. We then describe an image quality method that combines correlations of these spatial averages (“texture similarity”) with correlations of the feature maps (“structure similarity”). The parameters of the proposed measure are jointly optimized to match human ratings of image quality, while minimizing the reported distances between subimages cropped from the same texture images. Experiments show that the optimized method explains human perceptual scores, both on conventional image quality databases, as well as on texture databases. The measure also offers competitive performance on related tasks such as texture classification and retrieval. Finally, we show that our method is relatively insensitive to geometric transformations (e.g., translation and dilation), without use of any specialized training or data augmentation. Code is available at https://github.com/dingkeyan93/DISTS.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/f0f6998fb5a8f88c003860ca06f6ecaf6b5feec3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

TOPIQ: A Top-Down Approach From Semantics to Distortions for Image Quality Assessment (2023, methodology): “The same as [6], [35], we fitted a 4-parameter”
DiffIR: Efficient Diffusion Model for Image Restoration (2023, methodology): “Specifically, from level-1 to level-4, we set the number of dynamic transformer blocks to [1, 1, 1, 9], [13, 1, 1, 1], and [3, 5, 6, 6] for inpainting, SR, and deblurring, respectively.”
Dual Aggregation Transformer for Image Super-Resolution (2023, methodology): “To further evaluate our methods, we introduce two metrics: LPIPS [28] and DISTS [6].”
DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data (2023, methodology): “Further improvements include ensembling for robustness [44], antialiasing [23, 35, 89] for stability, and global descriptors for texture [23].”
Channel-Wise Autoregressive Entropy Models for Learned Image Compression (2020, background): “Interesting future research could look at combining our entropy model with more sophisticated image quality metrics such as perceptual metrics [33]– [35] and adversarial loss [36]–[38].”
Uncertainty-Aware Blind Image Quality Assessment in the Laboratory and Wild (2020, methodology): “We compare the performance of UNIQUE against three full-reference IQA measures - MS-SSIM [2], NLPD [40] and DISTS [41], and ten BIQA models, including four knowledge-driven methods that do not require MOSs for training - NIQE [10], IL-NIQE [26], dipIQ [15] and Ma19 [16], and six data-driven DNN-base”
No-Reference Image Quality Assessment via Transformers, Relative Ranking, and Self-Consistency (2021, methodology): “) is implemented via a Hamming window that approximately applies the Nyquist criterion [71].”
SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution (2023, neutral): “PSNR and SSIM [59] (calculated on the Y channel in YCbCr space) are reference-based fidelity measures, while LPIPS 3 [73], DISTS [13] are reference-based perceptual quality measures.”
PIPAL: a Large-Scale Image Quality Assessment Dataset for Perceptual Image Restoration (2020, methodology): “The anti-aliasing pooling layers (l2 pooling [11] and BlurPool [59]) improve the performance both on the PIPAL full set and GAN-based distortion subset based distortion.”
Pixel-Aware Stable Diffusion for Realistic Image Super-resolution and Personalized Stylization (2023, methodology): “It can be seen that our PASD method can generate more realistic details with better visual quality (see the synthesized textures in fur, flowers, leaves, feathers, etc.).”