Application of Just-Noticeable Difference in Quality as Environment Suitability Test for Crowdsourcing Speech Quality Assessment Task

By Babak Naderi, Sebastian Möller. 2020

20 citations recorded by the source corpus.

Crowdsourcing micro-task platforms facilitate subjective media quality assessment by providing access to a highly scaleable, geographically distributed and demographically diverse pool of crowd workers. Those workers participate in the experiment remotely from their own working environment, using their own hardware. In the case of speech quality assessment, preliminary work showed that environmental noise at the listener's side and the listening device (loudspeaker or headphone) significantly affect perceived quality, and consequently the reliability and validity of subjective ratings. As a consequence, ITU-T Rec. P.808 specifies requirements for the listening environment of crowd workers when assessing speech quality. In this paper, we propose a new Just Noticeable Difference of Quality (JNDQ) test as a remote screening method for assessing the suitability of the work environment for participating in speech quality assessment tasks. In a laboratory experiment, participants performed this JNDQ test with different listening devices in different listening environments, including a silent room according to ITU-T Rec. P.800 and a simulated background noise scenario. Results show a significant impact of the environment and the listening device on the JNDQ threshold. Thus, the combination of listening device and background noise needs to be screened in a crowdsourcing speech quality test. We propose a minimum threshold of our JNDQ test as an easily applicable screening method for this purpose.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

An Open source Implementation of ITU-T Recommendation P.808 with Validation (2020, methodology): “It reveals whether the working environment is suitable enough for participating in the test at that specific point of time [12].”
Subjective Evaluation of Noise Suppression Algorithms in Crowdsourcing (2021, methodology): “In the Setup section, usage of both ear-pods and the suitability of the participant’s environment are evaluated using a modiﬁed just-noticeable difference in quality test method [10].”
A Crowdsourcing Extension of the ITU-T Recommendation P.835 with Validation (2020, methodology): “In Setup section, usage of both ear-pods and suitability of the participant’s environment are evaluated using the modified just-noticeable difference in quality test method [4].”