Wave-U-Net: A Multi-Scale Neural Network for End-to-End Audio Source Separation

By Daniel Stoller, Sebastian Ewert, Simon E. Dixon. 2018

709 citations recorded by the source corpus.

Models for audio source separation usually operate on the magnitude spectrum, which ignores phase information and makes separation performance dependant on hyper-parameters for the spectral front-end. Therefore, we investigate end-to-end source separation in the time-domain, which allows modelling phase information and avoids fixed spectral transformations. Due to high sampling rates for audio, employing a long temporal input context on the sample level is difficult, but required for high quality separation results because of long-range temporal correlations. In this context, we propose the Wave-U-Net, an adaptation of the U-Net to the one-dimensional time domain, which repeatedly resamples feature maps to compute and combine features at different time scales. We introduce further architectural improvements, including an output layer that enforces source additivity, an upsampling technique and a context-aware prediction framework to reduce output artifacts. Experiments for singing voice separation indicate that our architecture yields a performance comparable to a state-of-the-art spectrogram-based U-Net architecture, given the same data. Finally, we reveal a problem with outliers in the currently used SDR evaluation metrics and suggest reporting rank-based statistics to alleviate this problem.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

Single Channel Voice Separation for Unknown Number of Speakers Under Reverberant and Noisy Settings (2020, methodology): “In this work, we focus on fully supervised voice separation using a single microphone, which has seen a great leap in performance following the recent success of deep learning models considering both frequency domain [1, 2, 7, 8, 9, 10], and time-domain [11, 12, 13, 14, 15, 16] modeling.”
Voice Separation with an Unknown Number of Multiple Speakers (2020, neutral): “We compared the proposed model to several highly competitive baselines; namely Demucs (Défossez et al., 2019), Wave-U-Net (Stoller et al., 2018), and Open-Unmix (St ”oter et al., 2019).”
Music Source Separation in the Waveform Domain (2019, methodology): “A convolutional network with a U-Net structure called Wave-U-Net was used ﬁrst on spectrograms [Jansson et al., 2017] and then adapted to the waveform domain [Stoller et al., 2018].”
Dual-Path RNN: Efficient Long Sequence Modeling for Time-Domain Single-Channel Speech Separation (2019, background): “Recent progress in deep learning-based speech separation has ignited the interest of the research community in time-domain approaches [1–6].”
FaSNet: Low-Latency Adaptive Beamforming for Multi-Microphone Audio Processing (2019, background): “The third category, which we refer to as the regression-based (RB) approach, implicitly incorporates beamforming within a neural network without explicitly generating the beamforming ﬁlters [26,27].”
HiFi++: a Unified Framework for Neural Vocoding, Bandwidth Extension and Speech Enhancement (2022, methodology): “The WaveUNet module is an instance of the well-known architecture Wave-U-Net (Stoller et al., 2018) which is a fully convolutional 1D-UNet-like neural network.”
Hybrid Transformers for Music Source Separation (2022, methodology): “Waveform based models started with Wave-U-Net [16], which served as the basis for Demucs [10], a time domain U-Net with a bi-LSTM between the encoder and decoder.”
Phase-aware Speech Enhancement with Deep Complex U-Net (2019, methodology): “Each of the corresponding time-frequency (t, f) representations computed by STFT is denoted as Xt,f ∈ C, Yt,f ∈ C, Zt,f ∈ C, and Ŷt,f ∈ C.”
DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement (2020, background): “The time-domain approaches can further fall into two categories — direct regression [2,3] and adaptive front-end approaches [4–6].”
Upsampling Artifacts in Neural Audio Synthesis (2020, methodology): “Unless using learnable interpolation ﬁlters [2], interpolation ﬁlters cannot be ﬁne-tuned and additional layers (like its subsequent learnable convolution) will have to compensate, if necessary, for the frequency response of the interpolation ﬁlter that introduces ﬁlter-ing artifacts.”
Singing Voice Separation: A Study on Training Data (2019, methodology): “Results are generally presented for a full procedure, including dataset building, data pre-processing and/or augmentation, architecture design, post-processing and sometimes a long engineering work to tune the hyperparameters of the models [7, 8, 5, 9].”
Wavelet Networks: Scale Equivariant Learning From Raw Waveforms (2020, background): “[64] Daniel Stoller, Sebastian Ewert, and Simon Dixon.”
Improving singing voice separation using Deep U-Net and Wave-U-Net with data augmentation (2019, methodology): “The Wave-U-Net has been trained on a rather limited dataset, consisting of the train part of the musdb18 dataset (≈6h) and the CCMixter dataset [9] (≈3h).”
Separate Anything You Describe (2023, neutral): “Time domain-based methods focus on mapping directly onto the audio waveform, such as WaveUNet [27], ConvTasNet [9], and Demucs [28].”
D3Net: Densely connected multidilated DenseNet for music source separation (2020, background): “The low-resolution representations are again upsampled to recover the lost resolution while carrying over the global perspective from downsampled layers [4, 5, 7, 10].”
Performance Study of a Convolutional Time-Domain Audio Separation Network for Real-Time Speech Denoising (2020, methodology): “Other time-domain methods have been recently proposed for audio source separation include U-Net inspired models such as Wave U-Net [12] and SEGAN [13].”