A scalable noisy speech dataset and online subjective test framework

By Chandan K. A. Reddy, Ebrahim Beyrami, Jamie Pool, Ross Cutler, Sriram Srinivasan, Johannes Gehrke. 2019

186 citations recorded by the source corpus.

Background noise is a major source of quality impairments in Voice over Internet Protocol (VoIP) and Public Switched Telephone Network (PSTN) calls. Recent work shows the efficacy of deep learning for noise suppression, but the datasets have been relatively small compared to those used in other domains (e.g., ImageNet) and the associated evaluations have been more focused. In order to better facilitate deep learning research in Speech Enhancement, we present a noisy speech dataset (MS-SNSD) that can scale to arbitrary sizes depending on the number of speakers, noise types, and Speech to Noise Ratio (SNR) levels desired. We show that increasing dataset sizes increases noise suppression performance as expected. In addition, we provide an open-source evaluation methodology to evaluate the results subjectively at scale using crowdsourcing, with a reference algorithm to normalize the results. To demonstrate the dataset and evaluation framework we apply it to several noise suppressors and compare the subjective Mean Opinion Score (MOS) with objective quality measures such as SNR, PESQ, POLQA, and VISQOL and show why MOS is still required. Our subjective MOS evaluation is the first large scale evaluation of Speech Enhancement algorithms that we are aware of.

Subfields: Artificial Intelligence, Biomedical Engineering, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Signal Processing

Citations in the corpus

Real Time Speech Enhancement in the Waveform Domain (2020, background): “Thus, speech enhancement is a particularly important task in itself for audio and video calls [1], hearing aids [2], and can also help automatic speech recognition (ASR) systems [3].”
BigVGAN: A Universal Neural Vocoder with Large-Scale Training (2022, methodology): “We simulate the unseen recording environment setup by adding a random environmental noise from MS-SNSD [46], such as airport, cafe, babble, etc.”
Dnsmos: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors (2020, methodology): “However, they are shown to correlate poorly with human rating when used for SE tasks that involve perceptually invariant transformations [1].”
ICASSP 2021 Deep Noise Suppression Challenge (2020, methodology): “We initially released MSSNSD [10] with a focus on extensibility, but the dataset lacked the diversity in speakers and noise types.”
Weighted Speech Distortion Losses for Neural-Network-Based Real-Time Speech Enhancement (2020, methodology): “Not only did all our selected systems significantly outperform the improved RNNoise (RNNoiseI) trained on MSE presented in [22], we were surprised that the listening test subjects preferred a rather low setting for the speech distortion weight α.”
Performance Study of a Convolutional Time-Domain Audio Separation Network for Real-Time Speech Denoising (2020, background): “However, the correlation of these metrics with subjective mean opinion score (MOS) is often not high enough to be reliable for many applications [23].”
WaveNetEQ — Packet Loss Concealment with WaveRNN (2020, methodology): “To make the model more robust we augment the dataset by creating synthetic mixes of the speech with background noise randomly selected from the MS-SNSD dataset[10].”
Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretraining (2020, methodology): “Besides the original ATIS, models are evaluated on its noisy copy (augmented with MS-SNSD [29]).”
Interspeech 2021 Deep Noise Suppression Challenge (2021, methodology): “We initially released MSSNSD [10] with a focus on extensibility, but the dataset lacked the diversity in speakers and noise types.”
Noise Robust TTS for Low Resource Speakers using Pre-trained Model and Speech Enhancement (2020, methodology): “In our experiments, four new speaker’s data is adopted, each speaker contains 200-300 utterances, and the low-quality data is constructed based on the Microsoft Scalable Noisy Speech Dataset [22], ensuring that the SNR of each utterance is less than 5dB.”
Dnsmos P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors (2021, methodology): “Subjective evaluation of speech quality is the most reliable way to evaluate Speech Enhancement (SE) methods [3].”
High Fidelity Speech Regeneration with Application to Speech Enhancement (2021, background): “PESQ validity for other tasks is questionable, e.g., it shows a low correlation with MOS [35].”
The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Speech Quality and Testing Framework (2020, methodology): “Previously, we released MSSNSD [13] with a focus on extensibility.”