Massive Online Crowdsourced Study of Subjective and Objective Picture Quality

By Deepti Ghadiyaram, A. Bovik. 2015

808 citations recorded by the source corpus.

Most publicly available image quality databases have been created under highly controlled conditions by introducing graded simulated distortions onto high-quality photographs. However, images captured using typical real-world mobile camera devices are usually afflicted by complex mixtures of multiple distortions, which are not necessarily well-modeled by the synthetic distortions found in existing databases. The originators of existing legacy databases usually conducted human psychometric studies to obtain statistically meaningful sets of human opinion scores on images in a stringently controlled visual environment, resulting in small data collections relative to other kinds of image analysis databases. Toward overcoming these limitations, we designed and created a new database that we call the LIVE In the Wild Image Quality Challenge Database, which contains widely diverse authentic image distortions on a large number of images captured using a representative variety of modern mobile devices. We also designed and implemented a new online crowdsourcing system, which we have used to conduct a very large-scale, multi-month image quality assessment (IQA) subjective study. Our database consists of over 350 000 opinion scores on 1162 images evaluated by over 8100 unique human observers. Despite the lack of control over the experimental environments of the numerous study participants, we demonstrate excellent internal consistency of the subjective data set. We also evaluate several top-performing blind IQA algorithms on it and present insights on how the mixtures of distortions challenge both end users as well as automatic perceptual quality prediction models. The new database is available for public use at <;uri xlink:href="http://live.ece.utexas.edu/research/ChallengeDB/index.html" xlink:type="simple">http://live.ece.utexas.edu/research/ChallengeDB/index.html<;/uri>.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

The Unreasonable Effectiveness of Deep Features as a Perceptual Metric (2018, background): “A related line of work is on No-Reference Image Quality Assessment (NR-IQA), such as AVA [38] and LIVE In the Wild [18].”
Uncertainty-Aware Blind Image Quality Assessment in the Laboratory and Wild (2020, methodology): “Database # of Images Scenario Annotation Range Subjective Testing Methodology LIVE [4] 779 Synthetic DMOS, Variance [0, 100] Single stimulus continuous quality rating CSIQ [5] 866 Synthetic DMOS, Variance [0, 1] Multi stimulus absolute category rating KADID-10K [6] 10,125 Synthetic MOS, Variance [1,”
Large-Scale Study of Perceptual Video Quality (2018, methodology): “This success of latter study [17] has inspired our work here, with a goal to build a large, diverse and representative video database on which we crowdsourced a large-scale subjective video quality study.”
Deep CNN-Based Blind Image Quality Predictor (2019, methodology): “proposed algorithm: LIVE IQA database [5], CSIQ [29], TID2013 [30], LIVE multiply distorted (LIVE MD) database [31], LIVE in the wild challenge (LIVE challenge) database [32], and Waterloo exploration database (WED) [33].”
Blind Image Quality Assessment Based on High Order Statistics Aggregation (2016, background): “LIVE In the Wild Image Quality Challenge Database (Challenge DB) [56]: This database contains 1162 test images”
MetaIQA: Deep Meta-Learning for No-Reference Image Quality Assessment (2020, methodology): “However, their application scope is rather limited, considering the fact that the distortion type is usually unknown in real applications [15, 9].”
KADID-10k: A Large-scale Artificially Distorted IQA Database (2019, neutral): “of Ratings Subjective study Database Year Content distorted images Distortion type distortion types rated images per image environment IVC [5] 2005 10 185 artificial 4 185 15 lab LIVE [6] 2006 29 779 artificial 5 779 23 lab CSIQ [7] 2009 30 866 artificial 6 866 5∼7 lab TID2013 [2] 2013 25 3,000 arti”
Perceptual Quality Assessment of Smartphone Photography (2020, methodology): “Ghadi-yaram and Bovik created the LIVE Challenge Database [5], which contains 1 , 162 images by 15 mobile cameras.”
Deep Neural Networks for No-Reference and Full-Reference Image Quality Assessment (2016, methodology): “and CSIQ [44] image quality databases, the NR IQA approach is also evaluated on the LIVE In the Wild Image Quality Challenge Database [22], [23].”
Blindly Assess Image Quality in the Wild Guided by a Self-Adaptive Hyper Network (2020, methodology): “The SRCC slightly improved on the LIVE Challenge Database and obviously improved on the LIVE Database with around 1.6% and 3%.”
From Patches to Pictures (PaQ-2-PiQ): Mapping the Perceptual Space of Picture Quality (2019, methodology): “This series of models obtains state-of-the art picture quality performance on the new database, and transfer well – without ﬁnetuning – on smaller “in-the-wild databases such as LIVE Challenge (CLIVE) [17] and KonIQ-10K [18] (Sec.”
Blind Image Quality Assessment Using a Deep Bilinear Convolutional Neural Network (2019, methodology): “LIVE Challenge [14] is an authentic IQA database, which contains 1, 162 images captured from diverse real-world scenes by numerous photographers with various levels of photography skills using different camera devices, and hence undergo complex realistic distortions.”
Perceptual quality prediction on authentically distorted images using a bag of features approach (2016, background): “The high internal statistical consistency and reliability of the subjective scores gathered in the crowdsource study make it possible to consider the MOS values obtained from the online study as ground truth quality scores of the images (Ghadiyaram & Bovik, 2016).”
Exploring CLIP for Assessing the Look and Feel of Images (2022, methodology): “However, these methods requires training with human-labeled benchmarks [8, 10, 15, 28, 39, 47], limiting their generalizability and versatility.”
UGC-VQA: Benchmarking Blind Video Quality Assessment for User Generated Content (2020, methodology): “The FRIQUEE model [39] has been observed to achieve SOTA performance both on UGC/consumer video/picture databases like LIVE-Challenge [14], CVD2014 [12], and KoNViD-1k [10] by leveraging a bag of NSS features drawn from diverse color spaces and perceptually motivated transform domains.”
The Konstanz natural video database (KoNViD-1k) (2017, background): “Recently, it has been shown that reliable measures of quality of experience can be generated by crowdsourcing for images [24], [3] and videos [25].”