Phase-sensitive and recognition-boosted speech separation using deep recurrent neural networks

By Hakan Erdogan, John R. Hershey, Shinji Watanabe, Jonathan Le Roux. 2015

674 citations recorded by the source corpus.

Separation of speech embedded in non-stationary interference is a challenging problem that has recently seen dramatic improvements using deep network-based methods. Previous work has shown that estimating a masking function to be applied to the noisy spectrum is a viable approach that can be improved by using a signal-approximation based objective function. Better modeling of dynamics through deep recurrent networks has also been shown to improve performance. Here we pursue both of these directions. We develop a phase-sensitive objective function based on the signal-to-noise ratio (SNR) of the reconstructed signal, and show that in experiments it yields uniformly better results in terms of signal-to-distortion ratio (SDR). We also investigate improvements to the modeling of dynamics, using bidirectional recurrent networks, as well as by incorporating speech recognition outputs in the form of alignment vectors concatenated with the spectral input features. Both methods yield further improvements, pointing to tighter integration of recognition with separation as a promising future direction.

Subfields: Cognitive Neuroscience, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Speech & Audio

In Signal Processing

Citations in the corpus

Light Gated Recurrent Units for Speech Recognition (2018, background): “Several works have already highlighted the effectiveness of RNNs in various speech processing tasks, such as speech recognition [35]–[39], speech enhancement [40], speech separation [41], [42] as well as speech activity detection [43].”
Time-Frequency Masking in the Complex Domain for Speech Dereverberation and Denoising (2017, positive): “Recently, a phase-sensitive mask (PSM) has been defined [8], which amounts to the real portion of the cIRM (i.”
Voice Separation with an Unknown Number of Multiple Speakers (2020, methodology): “A phase-sensitive objective function with an LSTM neural network was presented in (Erdogan et al., 2015), showing an improvement of SDR over the CHiME-2 (Vincent et al.”
Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation (2018, background): “Moreover, the separation accuracy of Conv-TasNet surpasses the performance of ideal time-frequency masks, including the ideal binary mask (IBM [28]), ideal ratio mask (IRM, [29], [30]), and Winener ﬁlter-like mask (WFM, [31]).”
Multitalker Speech Separation With Utterance-Level Permutation Invariant Training of Deep Recurrent Neural Networks (2017, background): “The Ideal Phase Sensitive Mask (IPSM) [43], [45] however, takes phase differences into consideration, where θ y and θ s are the phases of mixed speech Y ( t, f ) and source X s ( t, f ) , respectively.”
TaSNet: Time-Domain Audio Separation Network for Real-Time, Single-Channel Speech Separation (2017, background): “Even though several systems have been proposed to use the phase information to design the masks, such as the phase-sensitive mask [6] and complex ratio mask [7], the upper bound still exists since the reconstruction is not exact.”
Looking to listen at the cocktail party (2018, neutral): “Wang and Chen [2017] give a comprehensive overview of recent audio-only methods based on deep learning that tackle both speech denoising [Erdogan et al. 2015; Weninger et al. 2015] and speech separation tasks.”
SDR – Half-baked or Well Done? (2018, methodology): “Source separation and speech enhancement have been an intense focus of research in the signal processing community for several decades, and interest has gotten even stronger with the recent advent of powerful new techniques based on deep learning [1–11].”
End-to-End Speech Separation with Unfolded Iterative Phase Reconstruction (2018, methodology): “By Parseval’s theorem, this is equivalent to optimizing the mask for minimum squared error in the complex spectrum domain, when using the noisy phases, as in [21], proposed in the same conference.”
Fullsubnet: A Full-Band and Sub-Band Fusion Model for Real-Time Single-Channel Speech Enhancement (2020, methodology): “The frequency-domain methods [4–7] typically use the noisy spectral feature (e.g., complex spectrum, magnitude spectrum) as the input of a neural model.”
Speech Enhancement with LSTM Recurrent Neural Networks and its Application to Noise-Robust ASR (2015, methodology): “The D(R)NN topology and training parameters were set as in [16] and [2].”
The Phasebook: Building Complex Masks via Discrete Representations for Source Separation (2019, background): “We investigate the most popular magnitude masks, whose oracle performance when paired with the noisy phase was compared in [4]: ideal amplitude mask (IAM: a = |s| |x| ), here also considering its truncations to various thresh-”
Real-time Single-channel Dereverberation and Separation with Time-domain Audio Separation Network (2018, methodology): “Although there are methods such as phase-sensitive mask for separation [11] or complex ratio mask and iterative reconstruction for dereverberation [12, 7], the performance is still limited and model complexity might be much higher.”
Speech dereverberation and denoising using complex ratio masks (2017, methodology): “For each experiment, our proposed approach is compared to ideal ratio mask (IRM) estimation (denoted as RM) [19], phase sensitive mask (PSM) estimation [21], and direct estimation of spectral magnitudes (DSM) [6].”
Gated Residual Networks with Dilated Convolutions for Supervised Speech Separation (2018, positive): “Alternatively, one can estimate the phase-sensitive mask (PSM) [5], which yields a higher signal-to-noise ratio (SNR).”
WHAM!: Extending Speech Separation to Noisy Environments (2019, methodology): “…attempt to estimate a real-valued mask for each source ˆ M c ∈ R F × T by minimizing the truncated phase sensitive approximation (tPSA) objective [2] in a permutation-free manner [3,10,12]: where P is the set of all possible permutations over the set of sources { 1 , . . . , C } , (cid:12)…”
Divide and Conquer: A Deep CASA Approach to Talker-Independent Monaural Speaker Separation (2019, background): “Phase-sensitive approximation (PSA) is found to be effective as it accounts for errors introduced by noisy phase [8], [22].”
End-to-End Waveform Utterance Enhancement for Direct Evaluation Metrics Optimization by Fully Convolutional Neural Networks (2017, background): “The BLSTM had 2 bidirectional LSTM layers, each with 384 nodes as in [26] followed by a fully connected output layer.”