WHAM!: Extending Speech Separation to Noisy Environments

By Gordon Wichern, Joseph M. Antognini, Michael D. Flynn, Licheng Zhu, Emmett McQuinn, Dwight Crow, Ethan Manilow, Jonathan Le Roux. 2019

507 citations recorded by the source corpus.

Recent progress in separating the speech signals from multiple overlapping speakers using a single audio channel has brought us closer to solving the cocktail party problem. However, most studies in this area use a constrained problem setup, comparing performance when speakers overlap almost completely, at artificially low sampling rates, and with no external background noise. In this paper, we strive to move the field towards more realistic and challenging scenarios. To that end, we created the WSJ0 Hipster Ambient Mixtures (WHAM!) dataset, consisting of two speaker mixtures from the wsj0-2mix dataset combined with real ambient noise samples. The samples were collected in coffee shops, restaurants, and bars in the San Francisco Bay Area, and are made publicly available. We benchmark various speech separation architectures and objective functions to evaluate their robustness to noise. While separation performance decreases as a result of noise, we still observe substantial gains relative to the noisy signals for most approaches.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Yossi Adi
3. Shinji Watanabe
4. Dong Yu
5. Wei-Ning Hsu
6. Zhuo Chen
7. Lior Wolf
8. Shujie Liu

In Signal Processing

1. Jinyu Li
2. Yossi Adi
3. Shinji Watanabe
4. Zhuo Chen
5. Lior Wolf
6. Wei-Ning Hsu
7. Dong Yu
8. Mirco Ravanelli

Citations in the corpus

• SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2021, background): “Model WER Ground Truth Speech 3.2% Noisy Speech (Wichern et al., 2019) The results suggest that although the noisy speech with WHAM! is challenging as summarized in Table 12 (in Appendix B.5), the proposed encoder-decoder framework can effectively suppress the noise and recover the content.”
• Single Channel Voice Separation for Unknown Number of Speakers Under Reverberant and Noisy Settings (2020, methodology): “The clean signals were taken from the WSJ0 corpus [26] and noise signals from the WHAM! noise dataset [27].”
• Voice Separation with an Unknown Number of Multiple Speakers (2020, neutral): “(Wichern et al., 2019) and WHAMR!”
• LibriMix: An Open-Source Dataset for Generalizable Speech Separation (2020, unknown): “[9].”
• Generative Pre-training for Speech with Flow Matching (2023, methodology): “In addition, noise sampled from WHAM! dataset (Wichern et al., 2019) can be added to the mixture to further increase the difficulty of separation, combining 4 different setups in total.”
• Filterbank Design for End-to-end Speech Separation (2019, unknown): “In fact, the authors of [14] introduce a noisy extension of wsj0-2mix, WHAM, on which initial results indicate that the advantage of learned representations reduces as noise is introduced, suggesting that learning from the raw waveform might be harder in noisy conditions.”
• SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities (2022, unknown): “0 Licensed) and WHAM! (Wichern et al., 2019) (CC BY-NC 4.”
• Utilizing Self-supervised Representations for MOS Prediction (2021, methodology): “Additionally, we also collected real-world noises from WHAM! [21] and music from MUSAN [22].”
• A Survey of Sound Source Localization with Deep Learning Methods (2021, unknown): “I = 7 [103], I = 8 [116], I = 20 [169], I = 37 [127], I = 72 [58], I = 360 [101].”
• SpeechBrain: A General-Purpose Speech Toolkit (2021, methodology): “ConvTasNet[43] DualPath RNNs [44] SepFormer [45] WSJ-mix [46] WHAM [47] WHAMR [48] LibriMix [49] Spoken language understanding Speech to intent/slots.”
• AISHELL-4: An Open Source Dataset for Speech Enhancement, Separation, Recognition and Speaker Diarization in Conference Scenario (2021, methodology): “[14] G. Wichern, J. Antognini, M. Flynn, L. R. Zhu, E. McQuinn, D. Crow, E. Manilow, and J. L. Roux, “WHAM!”
• WHAMR!: Noisy and Reverberant Single-Channel Speech Separation (2019, methodology): “The STFT features are also chosen to be consistent with [11], with a window length of 32 ms and hop size of 8 ms.”
• Continuous Speech Separation: Dataset and Analysis (2020, background): “In [30], the authors collected replayed WSJ0 utterances in different outdoor environments to evaluate the noise robustness of speech separation systems.”
• RePlay: Retrieval-Based Voice Playback for Multi-Turn spoken dialogue (2026, unknown): “The 52,768 di-alogues (1071 h) are synthesised with Chatterbox TTS 2 using cloned VCTK [27] and LibriSpeech [28] voices, adding long user pauses and WHAM! noise [29].”