Universal Sound Separation

By Ilya Kavalerov, Scott Wisdom, Hakan Erdogan, Brian Patton, Kevin Wilson, Jonathan Le Roux, John R. Hershey. 2019

209 citations recorded by the source corpus.

Recent deep learning approaches have achieved impressive performance on speech enhancement and separation tasks. However, these approaches have not been investigated for separating mixtures of arbitrary sounds of different types, a task we refer to as universal sound separation, and it is unknown how performance on speech tasks carries over to non-speech tasks. To study this question, we develop a dataset of mixtures containing arbitrary sounds, and use it to investigate the space of mask-based separation architectures, varying both the overall network architecture and the framewise analysis-synthesis basis for signal transformations. These network architectures include convolutional long short-term memory networks and time-dilated convolution stacks inspired by the recent success of time-domain enhancement networks like ConvTasNet. For the latter architecture, we also propose novel modifications that further improve separation performance. In terms of the frame-wise analysis-synthesis basis, we explore both a short-time Fourier transform (STFT) and a learnable basis, as used in ConvTasNet. For both of these bases, we also examine the effect of window size. In particular, for STFTs, we find that longer windows (25-50 ms) work best for speech/non-speech separation, while shorter windows (2.5 ms) work best for arbitrary sounds. For learnable bases, shorter windows (2.5 ms) work best on all tasks. Surprisingly, for universal sound separation, STFTs outperform learnable bases. Our best methods produce an improvement in scale-invariant signal-to-distortion ratio of over 13 dB for speech/non-speech separation and close to 10 dB for universal sound separation.

Subfields: Cognitive Neuroscience, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Signal Processing

Citations in the corpus

SEANet: A Multi-modal Speech Enhancement Network (2020, methodology): “Namely, we include in our evaluation iTDCN++ [17] and Wavesplit [33].”
FSD50K: An Open Dataset of Human-Labeled Sound Events (2020, background): “Other tasks include search result clustering in large vocabulary datasets [98] or universal sound separation [99].”
Dual-Path RNN: Efficient Long Sequence Modeling for Time-Domain Single-Channel Speech Separation (2019, neutral): “The performance of the adaptive front-end methods also depend on selection of the window size, where a smaller window improves the separation performance at the cost of a significantly longer front-end representation [4, 11].”
TokenSplit: Using Discrete Speech Representations for Direct, Refined, and Transcript-Conditioned Speech Separation and Recognition (2023, methodology): “For the ﬁrst 3s of Libri2Mix test clips, we compare the separation performance of TokenSplit models to an audio-only TDCN++ model [24] in the top half of Table 1.”
Separate Anything You Describe (2023, unknown): “The work in [4] reported promising results on separating arbitrary sounds using permutation invariant training (PIT) [34], a supervised method initially designed for speech separation.”
Self-Supervised Learning from Automatically Separated Sound Scenes (2021, methodology): “The separation model used is based on an improved time-domain convolutional network (TDCN++) [20], similar to a Conv-TasNet [22].”
Improving Sound Event Detection in Domestic Environments using Sound Separation (2020, methodology): “The baseline SS model uses a similar architecture to an existing approach for universal sound separation with a fixed number of sources [9, 10], which employs a convolutional masking network using STFT and analysis and synthesis.”
Disentangling speech from surroundings in a neural audio codec (2022, background): “However, in contrast to conventional source separation and speech enhancement tasks [6, 7, 8], this separation happens in the embedding space.”
What’s all the Fuss about Free Universal Sound Separation Data? (2020, methodology): “hundreds of sound classes) universal sound separation shown to be feasible [1], though only with a fixed number of sources, and on proprietary data with restrictive licensing.”