DEEP CLUSTERING AND CONVENTIONAL NETWORKS FOR MUSIC SEPARATION: STRONGER TOGETHER

By Yi Luo, Zhuo Chen, John R. Hershey, Jonathan Le Roux, Nima Mesgarani. 2016

172 citations recorded by the source corpus.

Deep clustering is the first method to handle general audio separation scenarios with multiple sources of the same type and an arbitrary number of sources, performing impressively in speaker-independent speech separation tasks. However, little is known about its effectiveness in other challenging situations such as music source separation. Contrary to conventional networks that directly estimate the source signals, deep clustering generates an embedding for each time-frequency bin, and separates sources by clustering the bins in the embedding space. We show that deep clustering outperforms conventional networks on a singing voice separation task, in both matched and mismatched conditions, even though conventional networks have the advantage of end-to-end training for best signal approximation, presumably because its more flexible objective engenders better regularization. Since the strengths of deep clustering and conventional network architectures appear complementary, we explore combining them in a single hybrid network trained via an approach akin to multi-task learning. Remarkably, the combination significantly outperforms either of its components.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation (2018, background): “of STFT in most speech separation systems is at least 32 ms [5], [7], [8] and is even greater in music separation applications, which require an even higher resolution spectrogram (higher than 90 ms) [15], [16]. Because these issues arise from formulating the separation problem in the time-frequency”
TaSNet: Time-Domain Audio Separation Network for Real-Time, Single-Channel Speech Separation (2017, background): “Moreover, effective speech separation in STFT domain requires high frequency resolution which results in relatively large time window length, which is typically more than 32 ms for speech [3, 4, 5] and more than 90 ms for music separation [8].”
SDR – Half-baked or Well Done? (2018, methodology): “The scale-invariant SDR (SI-SDR) measure was used in [6,7,11,20–23].”
Wave-U-Net: A Multi-Scale Neural Network for End-to-End Audio Source Separation (2018, methodology): “To alleviate the problem of ﬁxed spectral representations widely used in previous work [6, 11, 13, 14, 20, 23], an adaptive front-end for spectrogram computation was developed [24] that is trained jointly with the separation network, which operates on the resulting magnitude spectrogram.”
End-to-End Speech Separation with Unfolded Iterative Phase Reconstruction (2018, methodology): “Following [22], our recent study [3] proposed a chimera++ network combining the two approaches via multi-task learning, as illustrated in the bottom of Fig.”
Singing Voice Separation with Deep U-Net Convolutional Networks (2017, unknown): “It should be noted that the Chimera web server is running an improved version of the algorithm that participated in MIREX, using a hybrid “multiple heads” architecture that combines deep clustering with a conventional neural network [16].”
Bootstrapping Single-channel Source Separation via Unsupervised Spatial Clustering on Stereo Mixtures (2018, background): “successful approach that has inspired multiple successful variants [22–27].”
SpeakerBeam: Speaker Aware Neural Network for Target Speaker Extraction in Speech Mixtures (2019, methodology): “We also explore the multi-task training of SpeakerBeam together with the Deep clustering method, in a similar fashion to that employed for Chimera networks [41], where the DC objective serves as a regularizer in a singing voice separation task.”
Universal Sound Separation (2019, background): “Singing voice separation has focused on vocal and music instrument tracks [15, 24].”
Singing Voice Separation: A Study on Training Data (2019, unknown): “In the previous literature [10, 11, 12, 13, 9] different architectures are usually compared using the same train/test datasets, but to the best of our knowledge, there are no previous works that study particularly the influence of these datasets.”
Speaker-Independent Speech Separation With Deep Attractor Network (2017, methodology): “Speciﬁcally, neural networks have been successfully applied in speech enhancement [6], [9], [10] and music separation [11], [12] with signiﬁcantly better performance compared with traditional methods.”