Music Source Separation in the Waveform Domain

By Alexandre D'efossez, Nicolas Usunier, L. Eon Bottou, Francis R. Bach. 2019

324 citations recorded by the source corpus.

Source separation for music is the task of isolating contributions, or stems, from different instruments recorded individually and arranged together to form a song. Such components include voice, bass, drums and any other accompaniments. Contrarily to many audio synthesis tasks where the best performances are achieved by models that directly generate the waveform, the state-of-the-art in source separation for music is to compute masks on the magnitude spectrum. In this paper, we first show that an adaptation of Conv-Tasnet (Luo & Mesgarani, 2019), a waveform-to-waveform model for source separation for speech, significantly beats the state-of-the-art on the MusDB dataset, the standard benchmark of multi-instrument source separation. Second, we observe that Conv-Tasnet follows a masking approach on the input signal, which has the potential drawback of removing parts of the relevant source without the capacity to reconstruct it. We propose Demucs, a new waveform-to-waveform model, which has an architecture closer to models for audio generation with more capacity on the decoder. Experiments on the MusDB dataset show that Demucs beats previously reported results in terms of signal to distortion ratio (SDR), but lower than Conv-Tasnet. Human evaluations show that Demucs has significantly higher quality (as assessed by mean opinion score) than Conv-Tasnet, but slightly more contamination from other sources, which explains the difference in SDR. Additional experiments with a larger dataset suggest that the gap in SDR between Demucs and Conv-Tasnet shrinks, showing that our approach is promising.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

High Fidelity Neural Audio Compression (2022, methodology): “Such modeling framework was shown to provide great results in various audio-related tasks, e.g., source separation and enhancement (Défossez et al., 2019; Defossez et al., 2020), neural vocoders (Kumar et al., 2019; Kong et al., 2020), audio codec (Zeghidour et al., 2021), and artiﬁcial bandwidth…”
Differentiable Model Compression via Pseudo Quantization Noise (2021, methodology): “2017) or source separation (Défossez et al. 2019), the typical model size is closer to 1GB, ruling out any kind of mobile usage.”
Real Time Speech Enhancement in the Waveform Domain (2020, methodology): “In this study we set f to be the DEMUCS architecture [11], which was initially developed for music source separation, and adapt it to the task of causal speech enhancement, a visual description of the model can be seen in Figure 1a.”
Voice Separation with an Unknown Number of Multiple Speakers (2020, methodology): “Unlike separating known sources (Défossez et al., 2019) in this case, the outputs are given in a permutation invariant fashion, hence, voices can switch between output-channels, especially during transient silence episodes.”
Upsampling Artifacts in Neural Audio Synthesis (2020, methodology): “23 - 10M Demucs [3]: transposed convolution (full-overlap) 5.”
Hybrid Transformers for Music Source Separation (2022, methodology): “Waveform based models started with Wave-U-Net [16], which served as the basis for Demucs [10], a time domain U-Net with a bi-LSTM between the encoder and decoder.”
Parallel and Flexible Sampling from Autoregressive Models via Langevin Dynamics (2021, methodology): “We compare PnF audio separation to results using the Demucs (Défossez et al., 2019) and Conv-Tasnet (Luo & Mesgarani, 2019) source separation models.”
HiFi++: a Unified Framework for Neural Vocoding, Bandwidth Extension and Speech Enhancement (2022, methodology): “For comparison with HiFi-GAN (Kong et al., 2020a), MelGAN (Kumar et al., 2019), Waveglow (Prenger et al., 2019), TFilm (Birnbaum et al., 2019), VoiceFixer (Liu et al., 2021), MetricGAN+ (Fu et al., 2021), DEMUCS (D´efossez et al., 2019) we employ the ofﬁcial implementations provided by the authors.”
Spleeter: a fast and efficient music source separation tool with pre-trained models (2020, background): “…Spatial distortion Ratio (ISR), are presented in the following table compared to Open-Unmix (Stöter, Uhlich, Liutkus, & Mitsufuji, 2019) and Demucs (Défossez, Usunier, Bottou, & Bach, 2019) (only SDR are reported for Demucs since other metrics are not available in the paper) which are, to the…”
Simple and Controllable Music Generation (2023, methodology): “To mitigate that, we used Demucs [Défossez et al., 2019] to ﬁrst decompose the reference track into four components: drums, bass, vocals, and other.”
D3Net: Densely connected multidilated DenseNet for music source separation (2020, background): “The low-resolution representations are again upsampled to recover the lost resolution while carrying over the global perspective from downsampled layers [4, 5, 7, 10].”