An Open source Implementation of ITU-T Recommendation P.808 with Validation

By Babak Naderi, Ross Cutler. 2020

110 citations recorded by the source corpus.

The ITU-T Recommendation P.808 provides a crowdsourcing approach for conducting a subjective assessment of speech quality using the Absolute Category Rating (ACR) method. We provide an open-source implementation of the ITU-T Rec. P.808 that runs on the Amazon Mechanical Turk platform. We extended our implementation to include Degradation Category Ratings (DCR) and Comparison Category Ratings (CCR) test methods. We also significantly speed up the test process by integrating the participant qualification step into the main rating task compared to a two-stage qualification and rating solution. We provide program scripts for creating and executing the subjective test, and data cleansing and analyzing the answers to avoid operational errors. To validate the implementation, we compare the Mean Opinion Scores (MOS) collected through our implementation with MOS values from a standard laboratory experiment conducted based on the ITU-T Rec. P.800. We also evaluate the reproducibility of the result of the subjective speech quality assessment through crowdsourcing using our implementation. Finally, we quantify the impact of parts of the system designed to improve the reliability: environmental tests, gold and trapping questions, rating patterns, and a headset usage test.

Subfields: Cognitive Neuroscience, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Speech & Audio

In Signal Processing

Citations in the corpus

Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, positive): “Speciﬁcally, we use a model trained with ground truth human ratings obtained using ITU-T P.808 [81] 6 The DNSMOS score predicted in the range of [1, 5], with a larger value indicating better quality of the input speech.”
Icassp 2022 Deep Noise Suppression Challenge (2022, methodology): “English clean speech is derived from Librivox 3 where we include audio clips chosen using a subjective ITU-T P.808 framework [7].”
Dnsmos: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors (2020, methodology): “A simple Convolutional Neural Network (CNN) based model is trained using the ground truth human ratings obtained using ITU-T P.808 [11, 12].”
DRED: Deep REDundancy Coding of Speech Using a Rate-Distortion-Optimized Variational Autoencoder (2022, methodology): “8 were obtained using the crowdsourcing methodology described in P.808 [37], [38], where each of the 966 test utterances was randomly assigned and evaluated by 10 naive listeners (not all utterances were rated by the same 10 listeners).”
ICASSP 2023 Deep Noise Suppression Challenge (2023, background): “(vi) Track-1 and Track-2 has their separate subjective evaluations to ensure we have headset testclips evaluated together, and speakerphone testclips evaluated together.”
Using IPA-Based Tacotron for Data Efficient Cross-Lingual Speaker Adaptation and Pronunciation Enhancement (2020, methodology): “We used the interface and analysis tools provided by [17] to acquire mean opinion scores (MOS) for our audio samples.”
Subjective Evaluation of Noise Suppression Algorithms in Crowdsourcing (2021, methodology): “We kept the structure of the P.808 Toolkit the same; further details about the P.808 Toolkit can be found in [8] and details on validation of the ITU-T Rec.”
A Crowdsourcing Extension of the ITU-T Recommendation P.835 with Validation (2020, methodology): “808 Toolkit [3] with methods for evaluating speech communication systems that include noise suppression algorithm 1.”
ICASSP 2022 Acoustic Echo Cancellation Challenge (2022, methodology): “We have extended the open source P.808 Toolkit [20] with methods for evaluating the echo impairments in subjective tests.”
Pretraining Techniques for Sequence-to-Sequence Voice Conversion (2020, methodology): “5) Subjective Evaluation Methods: The following subjective evaluations were performed using the open-source toolkit [79]”
NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasets (2021, methodology): “It is traditionally derived from subjective listening tests according to ITU-T P.800 [1] or recently also through crowdsourced listening tests according to ITU-T P.808 [2,3].”