No-Reference Image Quality Assessment via Transformers, Relative Ranking, and Self-Consistency

By S. Alireza Golestaneh, Saba Dadsetan, Kris M. Kitani. IEEE Workshop/Winter Conference on Applications of Computer Vision, 2021

429 citations recorded by the source corpus.

The goal of No-Reference Image Quality Assessment (NR-IQA) is to estimate the perceptual image quality in accordance with subjective evaluations, it is a complex and unsolved problem due to the absence of the pristine reference image. In this paper, we propose a novel model to address the NR-IQA task by leveraging a hybrid approach that benefits from Convolutional Neural Networks (CNNs) and self-attention mechanism in Transformers to extract both local and non-local features from the input image. We capture local structure information of the image via CNNs, then to circumvent the locality bias among the extracted CNNs features and obtain a non-local representation of the image, we utilize Transformers on the extracted features where we model them as a sequential input to the Transformer model. Furthermore, to improve the monotonicity correlation between the subjective and objective scores, we utilize the relative distance information among the images within each batch and enforce the relative ranking among them. Last but not least, we observe that the performance of NR-IQA models degrades when we apply equivariant transformations (e.g. horizontal flipping) to the inputs. Therefore, we propose a method that leverages self-consistency as a source of self-supervision to improve the robustness of NR-IQA models. Specifically, we enforce self-consistency between the outputs of our quality assessment model for each image and its transformation (horizontally flipped) to utilize the rich self-supervisory information and reduce the uncertainty of the model. To demonstrate the effectiveness of our work, we evaluate it on seven standard IQA datasets (both synthetic and authentic) and show that our model achieves state-of-the-art results on various datasets.1

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/16e3e5699b1389166b69717dcbbdead9d64e0da9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Guangtao Zhai
2. Xiaokang Yang
3. Weisi Lin
4. Kede Ma
5. Weixia Zhang
6. Yujiu Yang
7. Chaofeng Chen
8. Qiong Yan

In Media Technology

1. Weisi Lin
2. Guangtao Zhai
3. Xiaokang Yang
4. Kede Ma
5. Weixia Zhang
6. Haoning Wu
7. Ying Wei
8. Qiong Yan

Citations in the corpus

• MANIQA: Multi-dimension Attention Network for No-Reference Image Quality Assessment (2022, background): “Unfortunately, the internal disability of capturing non-local features and strong locality bias hinder the models to exploit the information across all regions of an image [9].”
• TOPIQ: A Top-Down Approach From Semantics to Distortions for Image Quality Assessment (2023, background): “They are generally improved with more advanced network architecture, from deep belief net [36] to CNN [37], then to deeper CNN [38], [39], [40], later to ResNet [41], [42], [43], and now vision transformers [44], [45], [46].”
• Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective (2023, methodology): “Being separately trained on each individual dataset with a separate group of parameters, DBCNN [76], HyperIQA [55], and TreS [13] achieve promising performance on all six datasets.”