End-to-End Speech Separation with Unfolded Iterative Phase Reconstruction

By Zhong-Qiu Wang, Jonathan Le Roux, Deliang Wang, John R. Hershey. 2018

134 citations recorded by the source corpus.

This paper proposes an end-to-end approach for single-channel speaker-independent multi-speaker speech separation, where time-frequency (T-F) masking, the short-time Fourier transform (STFT), and its inverse are represented as layers within a deep network. Previous approaches, rather than computing a loss on the reconstructed signal, used a surrogate loss based on the target STFT magnitudes. This ignores reconstruction error introduced by phase inconsistency. In our approach, the loss function is directly defined on the reconstructed signals, which are optimized for best separation. In addition, we train through unfolded iterations of a phase reconstruction algorithm, represented as a series of STFT and inverse STFT layers. While mask values are typically limited to lie between zero and one for approaches using the mixture phase for reconstruction, this limitation is less relevant if the estimated magnitudes are to be used together with phase reconstruction. We thus propose several novel activation functions for the output layer of the T-F masking, to allow mask values beyond one. On the publicly-available wsj0-2mix dataset, our approach achieves state-of-the-art 12.6 dB scale-invariant signal-to-distortion ratio (SI-SDR) and 13.1 dB SDR, revealing new possibilities for deep learning based phase reconstruction and representing a fundamental progress towards solving the notoriously-hard cocktail party problem.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

Voice Separation with an Unknown Number of Multiple Speakers (2020, methodology): “Wang et al. (2018c) propose a novel method to separate speech signals in the time-frequency domain.”
Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation (2018, methodology): “Although methods for phase reconstruction can be applied to alleviate this issue [11], [13], [14], the performance of the method remains suboptimal.”
SDR – Half-baked or Well Done? (2018, methodology): “Comparisons in [21] showed that there is a significant difference between SI-SDR and the SDR as implemented in BSS eval’s bss eval sources function.”
Dual-Path RNN: Efficient Long Sequence Modeling for Time-Domain Single-Channel Speech Separation (2019, neutral): “8 WA-MISI-5 [33] 32.”
WHAM!: Extending Speech Separation to Noisy Environments (2019, background): “This has brought forth a novel class of network architectures and objective functions beneﬁting from some type of phase processing, either implicitly by directly optimizing the time domain waveform [9, 13, 14], or explicitly via phase estimation algorithms [8, 13, 15].”
Deep Learning Based Phase Reconstruction for Speaker Separation: A Trigonometric Perspective (2018, methodology): “Following [6], we unfold the MISI iterations as multiple layers in the network and compute the loss function in the time domain.”
Recursive speech separation for unknown number of speakers (2019, methodology): “We chose a time domain approach as the phase reconstruction is shown to be important for source separation to improve the performance [11,22] and operating directly on time domain signals is possible.”
Learning to Optimize: A Primer and A Benchmark (2021, background): “Examples include clustering [221; 169; 170] and classification [222; 262; 59], phase retrieval [97], RNA second structure prediction [46], speech recognition and source separation [89; 225], remote sensing [130], smart grid [258], graph recovery [182], and photometric stereo estimation [235].”
Phase-aware Single-stage Speech Denoising and Dereverberation with U-Net (2020, methodology): “A time-domain loss function has recently been used as a popular loss function [6, 7, 8, 9, 10].”
Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processing (2019, methodology): “[35] 2018 Speech processing Source separation Multiple-input spectrogram inversion Adler et al.”
Divide and Conquer: A Deep CASA Approach to Talker-Independent Monaural Speaker Separation (2019, background): “In [41], a phase prediction network is further added to [40] to estimate the clean phase of each speaker source.”
A New Framework for CNN-Based Speech Enhancement in the Time Domain (2019, unknown): “Recently in [11], the authors integrate a fixed number of steps of the iterative Multiple Input Spectrogram Inversion (MISI) algorithm [12] inside DNN, which is found to be helpful for the speaker separation task.”
Wavesplit: End-to-End Speech Separation by Speaker Clustering (2020, unknown): “The separation stack is classical and resembles previous architectures conditioned on pre-trained speaker vectors (Wang et al., 2018a), or trained with PIT (Luo & Mes-garani, 2019).”
Differentiable Consistency Constraints for Improved Deep Speech Enhancement (2018, background): “[10] trained a real-valued masking-based system for speech separation through unfolded MISI iterations.”