Exploring CLIP for Assessing the Look and Feel of Images

By Jianyi Wang, Kelvin C. K. Chan, Chen Change Loy. AAAI Conference on Artificial Intelligence, 2022

1,372 citations recorded by the source corpus.

Measuring the perception of visual content is a long-standing problem in computer vision. Many mathematical models have been developed to evaluate the look or quality of an image. Despite the effectiveness of such tools in quantifying degradations such as noise and blurriness levels, such quantification is loosely coupled with human language. When it comes to more abstract perception about the feel of visual content, existing methods can only rely on supervised models that are explicitly trained with labeled data collected via laborious user study. In this paper, we go beyond the conventional paradigms by exploring the rich visual language prior encapsulated in Contrastive Language-Image Pre-training (CLIP) models for assessing both the quality perception (look) and abstract perception (feel) of images without explicit task-specific training. In particular, we discuss effective prompt designs and show an effective prompt pairing strategy to harness the prior. We also provide extensive experiments on controlled datasets and Image Quality Assessment (IQA) benchmarks. Our results show that CLIP captures meaningful priors that generalize well to different perceptual assessments.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/03ae89e796b1a8b566ae1554fab65c8c88b3a55f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild (2024, methodology): “We selected the following metrics for quantitative comparison: full-reference metrics PSNR, SSIM, LPIPS [93], and the non-reference metrics ManIQA [83], ClipIQA [74], MUSIQ [43].”
DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior (2023, methodology): “To better evaluate the quality , we include several no-reference image quality assessment (IQA) metrics: MANIQA[64], MUSIQ [24] and CLIP-IQA [51].”
Exploring Video Quality Assessment on User Generated Contents from Aesthetic and Technical Perspectives (2022, background): “with spatial [11], temporal [12], and semantic indices [55].”
Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective (2023, methodology): “We first (1) implement a pre-trained CLIP baseline which adopts a Likert-scale of two quality levels as in [62], i.”
SinSR: Diffusion-Based Image Super-Resolution in a Single Step (2023, methodology): “Besides, two recent SOTA non-reference metrics are used to justify the realism of all the images, i.e. , CLIP-IQA [39] which leverages a CLIP model [31] pre-trained on a large-scale dataset (Laion400M [36]) and MUSIQ [15].”
Exploiting Diffusion Prior for Real-World Image Super-Resolution (2023, neutral): “DIV2K Valid, RealSR and DRealSR, we employ various perceptual metrics including LPIPS [79], FID [25], CLIPIQA [60] and MUSIQ [32] to evaluate the perceptual quality of generated images.”
OminiControl: Minimal and Universal Control for Diffusion Transformer (2024, neutral): “Generation quality is measured using FID [16], SSIM, CLIP-IQA[53], MAN-IQA [59], MUSIQ [21], and PSNR[7] for visual fidelity, along with CLIP Text and CLIP Image scores [43] for consistency.”
ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shifting (2023, methodology): “4 Evaluation on Real-World Data Table 4 lists the comparative evaluation using CLIPIQA [66] and MUSIQ [65] of various methods on two real-world datasets.”
SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution (2023, neutral): “NIQE [71], MANIQA [64], MUSIQ [64], and CLIPIQA [64] are no-reference image quality measures.”