PIPAL: a Large-Scale Image Quality Assessment Dataset for Perceptual Image Restoration

By Jinjin Gu, Haoming Cai, Haoyu Chen, Xiaoxing Ye, Jimmy S. J. Ren, Chao Dong. 2020

298 citations recorded by the source corpus.

Image quality assessment (IQA) is the key factor for the fast development of image restoration (IR) algorithms. The most recent IR methods based on Generative Adversarial Networks (GANs) have achieved significant improvement in visual performance, but also presented great challenges for quantitative evaluation. Notably, we observe an increasing inconsistency between perceptual quality and the evaluation results. Then we raise two questions: (1) Can existing IQA methods objectively evaluate recent IR algorithms? (2) When focus on beating current benchmarks, are we getting better IR algorithms? To answer these questions and promote the development of IQA methods, we contribute a large-scale IQA dataset, called Perceptual Image Processing Algorithms (PIPAL) dataset. Especially, this dataset includes the results of GAN-based methods, which are missing in previous datasets. We collect more than 1.13 million human judgments to assign subjective scores for PIPAL images using the more reliable "Elo system". Based on PIPAL, we present new benchmarks for both IQA and super-resolution methods. Our results indicate that existing IQA methods cannot fairly evaluate GAN-based IR algorithms. While using appropriate evaluation methods is important, IQA methods should also be updated along with the development of IR algorithms. At last, we improve the performance of IQA networks on GAN-based distortions by introducing anti-aliasing pooling. Experiments show the effectiveness of the proposed method.

Subfields: Computer Vision and Pattern Recognition, Media Technology, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Chen Change Loy
3. Radu Timofte
4. Chao Dong
5. Yu Qiao
6. Lei Zhang
7. Ming-Hsuan Yang
8. Kai Zhang

In Media Technology

1. Radu Timofte
2. Chao Dong
3. Luc Van Gool
4. Chen Change Loy
5. Kai Zhang
6. Jingyun Liang
7. Jinjin Gu
8. Xintao Wang

In Evaluation & Benchmarks

1. Luc Van Gool
2. Kai Zhang
3. Chao Dong
4. Chen Change Loy
5. Yu Qiao
6. Jinjin Gu
7. Jingyun Liang
8. Xintao Wang

Citations in the corpus

• Designing a Practical Degradation Model for Deep Blind Image Super-Resolution (2021, methodology): “Such inconsistencies indicate that these no-reference IQA metrics do not always match perceptual visual quality [27] and the IQA metric could be updated with new SISR methods [14].”
• Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild (2024, background): “This phenomenon has also been noted in many studies as well [6, 26, 28].”
• Interpreting Super-Resolution Networks with Local Attribution Maps (2020, methodology): “[19] and employ both PSNR and the LPIPS perceptual similarity [58].”
• Exploring CLIP for Assessing the Look and Feel of Images (2022, methodology): “The main limitation of FR-IQA is that it is confined to scenarios where references exist.”
• TOPIQ: A Top-Down Approach From Semantics to Distortions for Image Quality Assessment (2023, methodology): “For FR datasets, we have LIVE [75], CSIQ [15], TID2013 [1], KADID-10k [76], PieAPP [22], BAPPS [8] and PIPAL [23].”
• MANIQA: Multi-dimension Attention Network for No-Reference Image Quality Assessment (2022, methodology): “PIPAL [13] is a recently proposed IQA dataset, which contains images processed by image restoration and enhancement methods (particularly the deep learning-based methods) besides the traditional distorting methods.”