End-To-End Source Separation With Adaptive Front-Ends

By Shrikant Venkataramani, Paris Smaragdis. 2017

77 citations recorded by the source corpus.

Source separation and other audio applications have traditionally relied on the use of short-time Fourier transforms as a front-end frequency domain representation step. The unavailability of a neural network equivalent to forward and inverse transforms hinders the implementation of end-to-end learning systems for these applications. We develop an auto-encoder neural network that can act as an equivalent to short-time front-end transforms. We demonstrate the ability of the network to learn optimal, real-valued basis functions directly from the raw waveform of a signal and further show how it can be used as an adaptive front-end for supervised source separation. In terms of separation performance, these transforms significantly outperform their Fourier counterparts. Finally, we also propose and interpret a novel source to distortion ratio based cost function for end-to-end source separation.

Subfields: Computational Mechanics, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Speech & Audio

In Signal Processing

Citations in the corpus

A Wavenet for Speech Denoising (2017, neutral): “Results are in line with recent work demonstrating that end-to-end pipelines are starting to prove effective in source separation tasks [33].”
Single Channel Voice Separation for Unknown Number of Speakers Under Reverberant and Noisy Settings (2020, methodology): “In this work, we focus on fully supervised voice separation using a single microphone, which has seen a great leap in performance following the recent success of deep learning models considering both frequency domain [1, 2, 7, 8, 9, 10], and time-domain [11, 12, 13, 14, 15, 16] modeling.”
Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation (2018, methodology): “In [19], a convolutional encoder motivated by discrete cosine transform (DCT) is used as the front-end.”
Wave-U-Net: A Multi-Scale Neural Network for End-to-End Audio Source Separation (2018, methodology): “To alleviate the problem of ﬁxed spectral representations widely used in previous work [6, 11, 13, 14, 20, 23], an adaptive front-end for spectrogram computation was developed [24] that is trained jointly with the separation network, which operates on the resulting magnitude spectrogram.”
Dual-Path RNN: Efficient Long Sequence Modeling for Time-Domain Single-Channel Speech Separation (2019, methodology): “Separation is applied to the front-end output as with the conventional time-frequency domain methods applying the separation processes to spectrogram inputs [3–5].”
End-to-End Speech Separation with Unfolded Iterative Phase Reconstruction (2018, background): “Closest to the above WA objective, an adaptive front-end framework was recently proposed [26] in which the STFT and its inverse are subsumed by the network, along with the noisy phase, so that training is effectively end-to-end in the time-domain.”
Attention Is All You Need In Speech Separation (2020, unknown): “The proposed model is based on the learned-domain masking approach [14, 15, 17–22] and employs an encoder, a decoder, and a masking network, as shown in Figure 1.”
End-to-end music source separation: is it possible in the waveform domain? (2018, methodology): “Finally, as an additional way to validate the direction we explored, it is worth mentioning that the speech source separation community is also starting to propose end-to-end methods with some degree of success [29, 30, 31].”