TaSNet: Time-Domain Audio Separation Network for Real-Time, Single-Channel Speech Separation

By Yi Luo, Nima Mesgarani. 2017

737 citations recorded by the source corpus.

Robust speech processing in multi-talker environments requires effective speech separation. Recent deep learning systems have made significant progress toward solving this problem, yet it remains challenging particularly in real-time, short latency applications. Most methods attempt to construct a mask for each source in time-frequency representation of the mixture signal which is not necessarily an optimal representation for speech separation. In addition, time-frequency decomposition results in inherent problems such as phase/magnitude decoupling and long time window which is required to achieve sufficient frequency resolution. We propose Time-domain Audio Separation Network (TasNet) to overcome these limitations. We directly model the signal in the time-domain using an encoder-decoder framework and perform the source separation on nonnegative encoder outputs. This method removes the frequency decomposition step and reduces the separation problem to estimation of source masks on encoder outputs which is then synthesized by the decoder. Our system outperforms the current state-of-the-art causal and noncausal speech separation algorithms, reduces the computational cost of speech separation, and significantly reduces the minimum required latency of the output. This makes TasNet suitable for applications where low-power, real-time implementation is desirable such as in hearable and telecommunication devices.

Subfields: Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Speech & Audio

In Signal Processing

Citations in the corpus

SEANet: A Multi-modal Speech Enhancement Network (2020, background): “Speech denoising can also be seen as a special case of source separation, in which one of the sources represents the speech signal of interest [14, 15, 16, 17].”
Speech Denoising with Deep Feature Losses (2018, background): “sults in speech processing [19, 20] including speech denoising [21, 22, 23, 24].”
Voice Separation with an Unknown Number of Multiple Speakers (2020, methodology): “We compare with the following baseline methods: ADANet (Luo et al., 2018), DPCL++ (Isik et al., 2016), CBLDNN-GAT (Li et al., 2018), TasNet (Luo & Mesgarani, 2018), the Ideal Ratio Mask (IRM), the Ideal Binary Mask (IBM), ConvTasNet (Luo & Mesgarani, 2019), FurcaNeXt (Zhang et al., 2020), and DPRNN ”
Music Source Separation in the Waveform Domain (2019, methodology): “The original Conv-Tasnet architecture Conv-Tasnet [Luo and Mesgarani, 2018] is composed of a learnt front-end that transforms back and forth between the input monophonic mixture waveform sampled at 8 kHz and a 128 channels over-complete representation sampled at 1 kHz using a convolution as the…”
Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation (2018, methodology): “ncy approaches, particularly in terms of their ability to scale and generalize to large data. On the other hand, a few recent studies have explored deep learning for time-domain audio separation [19]–[21]. The shared idea in all arXiv:1809.07454v3 [cs.SD] 15 May 2019 2 these systems is to replace th”
SDR – Half-baked or Well Done? (2018, methodology): “Both SI-SDR and BSS eval ’s SDR have recently been used by various studies [6–9,11,21–23,25,26] in the context of single-channel speaker-independent speech separation on the wsj0-2mix dataset [6], some of these studies reporting both ﬁgures [21–23 ,25].”
Wave-U-Net: A Multi-Scale Neural Network for End-to-End Audio Source Separation (2018, methodology): “…model performs a spectral analysis, and how to incorporate computations similar to those in a multi-scale ﬁlterbank, or to explicitly compute a decomposition of the input signal into a hierarchical set of basis signals and weightings on which to perform the separation, similar to the TasNet [12].”
Dual-Path RNN: Efficient Long Sequence Modeling for Time-Domain Single-Channel Speech Separation (2019, methodology): “Although DPRNN can be applied to any systems that require longterm sequential modeling, we investigate its application to the timedomain audio separation network (TasNet) [1, 4, 22], an adaptive front-end method that achieves high speech separation performance on a benchmarking dataset.”
End-To-End Source Separation With Adaptive Front-Ends (2017, methodology): “Common approaches include neural networks which given the noisy magnitudes, either predict a noiseless magnitude spectrum [5, 6, 7], or some type of a masking function [8, 9].”
Continuous Speech Separation: Dataset and Analysis (2020, background): “On top of, or aside from, these algorithmic improvements, researchers also sought more effective input features [17, 18] and model architectures [19, 20, 21].”
LibriMix: An Open-Source Dataset for Generalizable Speech Separation (2020, background): “To date, state-of-the-art performance on the single-channel speech separation task is achieved by deep learning based models [2–6].”
Exploring the Best Loss Function for DNN-Based Low-latency Speech Enhancement with Temporal Convolutional Networks (2020, background): “In particular, the Time-domain audio separation network (TasNet) [14] and its fully convolutional version (Conv-TasNet) [15], which have a trainable encoder-decoder architecture and do not depend on the STFT, have surpassed the performance of the ideal binary mask (IBM) for speech separation; in oth”
Deep Learning Based Phase Reconstruction for Speaker Separation: A Trigonometric Perspective (2018, neutral): “8 - - - - conv-TasNet-gLN [40], [38] 14.”
Performance Study of a Convolutional Time-Domain Audio Separation Network for Real-Time Speech Denoising (2020, methodology): “It should be noted that SI-SAR was not considered in the original Tas-Net papers [10, 11], and that the usefulness of separating out SAR from SDR has been questioned [19].”
Real-time Single-channel Dereverberation and Separation with Time-domain Audio Separation Network (2018, methodology): “We also apply the curriculum training strategy [17] in a similar fashion to [4].”
End-to-end music source separation: is it possible in the waveform domain? (2018, methodology): “Finally, as an additional way to validate the direction we explored, it is worth mentioning that the speech source separation community is also starting to propose end-to-end methods with some degree of success [29, 30, 31].”
Recursive speech separation for unknown number of speakers (2019, methodology): “For the error, in this work, we use the scaleinvariant signal-to-noise ratio (SI-SNR)(1), which has successfully been used in speech separation in the literature [10,15,16].”