Subjective Evaluation of Noise Suppression Algorithms in Crowdsourcing

By Babak Naderi, Ross Cutler. 2021

31 citations recorded by the source corpus.

The quality of the speech communication systems, which include noise suppression algorithms, are typically evaluated in laboratory experiments according to the ITU-T Rec. P.835, in which participants rate background noise, speech signal, and overall quality separately. This paper introduces an open-source toolkit for conducting subjective quality evaluation of noise suppressed speech in crowdsourcing. We followed the ITU-T Rec. P.835, and P.808 and highly automate the process to prevent moderator's error. To assess the validity of our evaluation method, we compared the Mean Opinion Scores (MOS), calculate using ratings collected with our implementation, and the MOS values from a standard laboratory experiment conducted according to the ITU-T Rec P.835. Results show a high validity in all three scales namely background noise, speech signal and overall quality (average PCC = 0.961). Results of a round-robin test (N=5) showed that our implementation is also a highly reproducible evaluation method (PCC=0.99). Finally, we used our implementation in the INTERSPEECH 2021 Deep Noise Suppression Challenge as the primary evaluation metric, which demonstrates it is practical to use at scale. The results are analyzed to determine why the overall performance was the best in terms of background noise and speech quality.

Subfields: Electrical and Electronic Engineering, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Electrical and Electronic Engineering

In Signal Processing

Citations in the corpus

Icassp 2022 Deep Noise Suppression Challenge (2022, methodology): “This challenge relies on ITU-T P.835 [8] subjective evaluation for evaluating the DNS models, since objective speech quality metrics, such as PESQ [22], SDR, and POLQA [23], do not correlate well with subjective speech quality [24].”
Dnsmos P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors (2021, methodology): “Naderi and Cutler [9] used this linear relationship to analyze the results of the 3rd Deep Noise Suppression challenge [10] to estimate the potential improvement in OVRL given a noise suppressor that maximized BAK.”
FRCRN: Boosting Feature Representation Using Frequency Recurrence for Monaural Speech Enhancement (2022, unknown): “We use the same model to enhance the blind test set and Table 5 shows the DNS-2022 P.835 [29] subjective evaluation results and WAcc re-sults.”
ICASSP 2022 Acoustic Echo Cancellation Challenge (2022, methodology): “The 1st place model [26] also won the ICASSP 2022 DNS Challenge [27], providing the only model that didn’t induce SIG [28] distortion in that challenge.”