Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation
By Yi Luo, Nima Mesgarani. 2018
2,194 citations recorded by the source corpus.
Single-channel, speaker-independent speech separation methods have recently seen great progress. However, the accuracy, latency, and computational cost of such methods remain insufficient. The majority of the previous methods have formulated the separation problem through the time–frequency representation of the mixed signal, which has several drawbacks, including the decoupling of the phase and magnitude of the signal, the suboptimality of time–frequency representation for speech separation, and the long latency in calculating the spectrograms. To address these shortcomings, we propose a fully convolutional time-domain audio separation network (Conv-TasNet), a deep learning framework for end-to-end time-domain speech separation. Conv-TasNet uses a linear encoder to generate a representation of the speech waveform optimized for separating individual speakers. Speaker separation is achieved by applying a set of weighting functions (masks) to the encoder output. The modified encoder representations are then inverted back to the waveforms using a linear decoder. The masks are found using a temporal convolutional network consisting of stacked one-dimensional dilated convolutional blocks, which allows the network to model the long-term dependencies of the speech signal while maintaining a small model size. The proposed Conv-TasNet system significantly outperforms previous time–frequency masking methods in separating two- and three-speaker mixtures. Additionally, Conv-TasNet surpasses several ideal time–frequency magnitude masks in two-speaker speech separation as evaluated by both objective distortion measures and subjective quality assessment by human listeners. Finally, Conv-TasNet has a significantly smaller model size and a shorter minimum latency, making it a suitable solution for both offline and real-time speech separation applications. This study, therefore, represents a major step toward the realization of speech separation systems for real-world speech processing technologies.
Subfields: Artificial Intelligence, Speech & Audio, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Speech & Audio
- Shinji Watanabe
- Yossi Adi
- Jinyu Li
- Wei-Ning Hsu
- John R. Hershey
- Deliang Wang
- Gabriel Synnaeve
- Zhuo Chen
In Signal Processing
Citations in the corpus
- Noise Estimation for Generative Diffusion Models (2021, methodology): “We trained a small model Pθ based on a the version of ConvTASNet [21] architecture.”
- High Fidelity Neural Audio Compression (2022, methodology): “For objective metrics, we use ViSQOL (Hines et al., 2012; Chinen et al., 2020) 2 , together with the Scale-Invariant Signal-to-Noise Ration (SI-SNR) (Luo & Mesgarani, 2019; Nachmani et al., 2020; Chazan et al., 2021).”
- UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, background): “…TSE, TT-Music, VC, S-Edit, SD, SVS: (1) their neural network architectures are based on transformer (Ren et al., 2020) or others (Oord et al., 2016; Luo & Mesgarani, 2019); (2) their training objectives can be either in time-domain (Luo & Mesgarani, 2019), frequency-domain (Yu et al., 2017) or…”
- Generative Speech Coding with Predictive Variance Regularization (2021, methodology): “To avoid the need for modeling summed independent signals, the input is pre-processed with a real-time TasNet [20,21].”
- End-to-End Neural Speech Coding for Real-Time Communications (2022, positive): “The TCM module as that in [14] consists of two convolutions with a kernel size of 1 × 1 to change channel dimensions and dilated depthwise convolu-tions to exploit temporal correlations with low complexity.”
- Single Channel Voice Separation for Unknown Number of Speakers Under Reverberant and Noisy Settings (2020, methodology): “We compared the proposed method to ConvTasNet [11], Dual-Path RNN (DPRNN) [14], and Gated model [22], for the case of a known number of speakers.”
- Voice Separation with an Unknown Number of Multiple Speakers (2020, methodology): “Then, Luo & Mesgarani (2019) suggested to replace the separator network from LSTM to a fully convolutional model using block of time depth separable dilated convolution (Conv-Tasnet).”
- Music Source Separation in the Waveform Domain (2019, background): “Thanks to its dilated convolutions with a stride of 1, the mask predictor of Conv-Tasnet is naturally time equivariant and even if the encoder/decoder is not strictly equivariant, Conv-Tasnet still verifies this property experimentally [Luo and Mesgarani, 2019].”
- Dual-Path RNN: Efficient Long Sequence Modeling for Time-Domain Single-Channel Speech Separation (2019, methodology): “Although DPRNN can be applied to any systems that require longterm sequential modeling, we investigate its application to the timedomain audio separation network (TasNet) [1, 4, 22], an adaptive front-end method that achieves high speech separation performance on a benchmarking dataset.”
- Fullsubnet: A Full-Band and Sub-Band Fusion Model for Real-Time Single-Channel Speech Enhancement (2020, methodology): “In the real-time inference stage, we usually use the cumulative normalization method [18, 19], i.e., at each time, the mean value used for normalization is computed using all the available frames.”
- Towards Neural Diarization for Unlimited Numbers of Speakers Using Global and Local Attractors (2021, background): “Some neural-network-based speech separation methods [19, 39, 40, 41, 42] limit the number of outputs by their network architecture, and thus, there is no way to deal with the mismatch.”
- FullSubNet+: Channel Attention Fullsubnet with Complex Spectrograms for Speech Enhancement (2022, methodology): “The “G-norm” is a global layer normalization [17].”
- MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancement (2019, background): “In recent years, various deep learning-based models have been developed for SE (Lu et al., 2013; Xu et al., 2014; Wang et al., 2014; Xu et al., 2015; Ochiai et al., 2017; Luo & Mesgarani, 2018; Grais et al., 2018; Germain et al., 2018; Chai et al., 2018; Choi et al., 2019).”
- Attention Is All You Need In Speech Separation (2020, neutral): “Model SI-SNRi SDRi # Param ConvTasnet [15] 12.”
- Dynamical Variational Autoencoders: A Comprehensive Review (2020, background): “In particular, temporal convolutional networks (TCNs) are getting more and more popular due to competitive performance with RNNs (e.g., in speech separation (Luo and Mesgarani, 2019)), while being more flexible and computationally efficient (Bai et al., 2018).”
- SpeechBrain: A General-Purpose Speech Toolkit (2021, methodology): “The mask encoded representations are then converted back to the waveforms using a linear decoder [43].”
- Don’t Shoot Butterfly with Rifles: Multi-Channel Continuous Speech Separation with Early Exit Transformer (2020, methodology): “Recently, the deep learning methods have been rigorously explored for better speech separation capability, including dual-path RNN [7], Conv-tasnet [8], and deep CASA [9] employing RNN and CNN structures.”
- TokenSplit: Using Discrete Speech Representations for Direct, Refined, and Transcript-Conditioned Speech Separation and Recognition (2023, background): “Many methods have been proposed recently that typically involve training a neural network on artificial mixtures of isolated speech to estimate individual sources [1, 2, 3].”