Dual-Path RNN: Efficient Long Sequence Modeling for Time-Domain Single-Channel Speech Separation

By Yi Luo, Zhuo Chen, Takuya Yoshioka. 2019

972 citations recorded by the source corpus.

Recent studies in deep learning-based speech separation have proven the superiority of time-domain approaches to conventional time-frequency-based methods. Unlike the time-frequency domain approaches, the time-domain separation systems often receive input sequences consisting of a huge number of time steps, which introduces challenges for modeling extremely long sequences. Conventional recurrent neural networks (RNNs) are not effective for modeling such long sequences due to optimization difficulties, while one-dimensional convolutional neural networks (1-D CNNs) cannot perform utterance-level sequence modeling when its receptive field is smaller than the sequence length. In this paper, we propose dual-path recurrent neural network (DPRNN), a simple yet effective method for organizing RNN layers in a deep structure to model extremely long sequences. DPRNN splits the long sequential input into smaller chunks and applies intra- and inter-chunk operations iteratively, where the input length can be made proportional to the square root of the original sequence length in each operation. Experiments show that by replacing 1-D CNN with DPRNN and apply sample-level modeling in the time-domain audio separation network (TasNet), a new state-of-the-art performance on WSJ0-2mix is achieved with a 20 times smaller model than the previous best system.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

Single Channel Voice Separation for Unknown Number of Speakers Under Reverberant and Noisy Settings (2020, methodology): “We compared the proposed method to ConvTasNet [11], Dual-Path RNN (DPRNN) [14], and Gated model [22], for the case of a known number of speakers.”
Voice Separation with an Unknown Number of Multiple Speakers (2020, methodology): “Recent studies pointed out the importance of choosing small kernel size for the encoder (Luo et al., 2019c).”
Music Source Separation in the Waveform Domain (2019, background): “[ Luo et al., 2020] introduced the Dual-Path RNN architecture for speech separation.”
Harmonic Gated Compensation Network Plus for ICASSP 2022 DNS Challenge (2022, methodology): “Between the encoder and decoder, DPRNN [12, 13] is inserted to model the multidimensional dependencies and Skip Connection concatenates the output of each encoder to the input of the corresponding decoder (red line in Fig.”
SpeechBrain: A General-Purpose Speech Toolkit (2021, methodology): “Dual-Path Speech separation Splits long speech inputs into smaller chunks and applies intra-and inter-chunk operations over them [44].”
Don’t Shoot Butterfly with Rifles: Multi-Channel Continuous Speech Separation with Early Exit Transformer (2020, methodology): “Recently, the deep learning methods have been rigorously explored for better speech separation capability, including dual-path RNN [7], Conv-tasnet [8], and deep CASA [9] employing RNN and CNN structures.”
Discretization and Re-synthesis: an alternative method to solve the Cocktail Party Problem (2021, methodology): “In our experiments, we adopt the same architecture of the DPRNN model [6] to better compare our proposed method and existing ones.”
Continuous Speech Separation: Dataset and Analysis (2020, background): “3 dB while [24] improved the SDR by 19.”
Torchaudio-Squim: Reference-Less Speech Quality and Intelligibility Measures in Torchaudio (2023, methodology): “√ As in [25], the value of R is selected such that R ≈ 2 L ≈ S for the 5-second training signals.”
FB-MSTCN: A Full-Band Single-Channel Speech Enhancement Method Based on Multi-Scale Temporal Convolutional Network (2022, methodology): “Although end-to-end modeling schemes in the time domain, such as Conv-TasNet [8] and DPRNN [9], can not be affected by the input feature dimension, it will increase the number of processing frames, resulting in a threefold increase in the number of operations to process audio per second.”
LibriMix: An Open-Source Dataset for Generalizable Speech Separation (2020, background): “Conv-TasNet [4], Dual-path RNN [5] or Wavesplit [6]) perform so well in separating fully overlapping speech mixtures from the wsj0-2mix dataset [2] that the separated speech estimates are almost indistinguishable from the reference signals.”
Multi-talker ASR for an unknown number of sources: Joint training of source counting, separation and ASR (2020, methodology): “It was shown in [15] to be superior to a separator based on 1-D convolutions used in the Conv-TasNet [24].”
Sequence to Multi-Sequence Learning via Conditional Chain Mapping for Mixture Signals (2020, background): “Speech Separation As the core part of the cocktail party problem [7], speech separation draws much attention recently [13, 15, 51, 19, 23, 20, 21, 24].”
Continuous Speech Separation with Conformer (2020, background): “The network architecture has been rigorously explored for better separation capability [16, 23, 24].”
Wavesplit: End-to-End Speech Separation by Speaker Clustering (2020, positive): “This is particularly remarkable as Wavesplit is trained on only 1s long windows, unlike ConvTasNet and Dual-Path RNN that are both trained on 4s long windows.”