Complex Ratio Masking for Monaural Speech Separation

By Donald S. Williamson, Yuxuan Wang, Deliang Wang. 2015

820 citations recorded by the source corpus.

Speech separation systems usually operate on the short-time Fourier transform (STFT) of noisy speech, and enhance only the magnitude spectrum while leaving the phase spectrum unchanged. This is done because there was a belief that the phase spectrum is unimportant for speech enhancement. Recent studies, however, suggest that phase is important for perceptual quality, leading some researchers to consider magnitude and phase spectrum enhancements. We present a supervised monaural speech separation approach that simultaneously enhances the magnitude and phase spectra by operating in the complex domain. Our approach uses a deep neural network to estimate the real and imaginary components of the ideal ratio mask defined in the complex domain. We report separation results for the proposed method and compare them to related systems. The proposed approach improves over other methods when evaluated with several objective metrics, including the perceptual evaluation of speech quality (PESQ), and a listening test where subjects prefer the proposed approach with at least a 69% rate.

Subfields: Cognitive Neuroscience, Computational Mechanics, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computational Mechanics

In Signal Processing

Citations in the corpus

Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis (2023, background): “Researchers have explored the use of deep learning for directly modeling the phase spectrum, but this remains a challenging area (Williamson et al., 2015).”
Time-Frequency Masking in the Complex Domain for Speech Dereverberation and Denoising (2017, positive): “A variant of this feature set has been shown to be effective for speech separation [38], and they have recently been shown to work well for cIRM estimation [42].”
Multitalker Speech Separation With Utterance-Level Permutation Invariant Training of Deep Recurrent Neural Networks (2017, background): “This is because phase estimation is still an open problem in the speech separation setup [42], [43].”
TaSNet: Time-Domain Audio Separation Network for Real-Time, Single-Channel Speech Separation (2017, background): “Even though several systems have been proposed to use the phase information to design the masks, such as the phase-sensitive mask [6] and complex ratio mask [7], the upper bound still exists since the reconstruction is not exact.”
End-to-End Speech Separation with Unfolded Iterative Phase Reconstruction (2018, background): “[27, 28] proposed a complex ratio masking approach for phase reconstruction and speech enhancement, where a feed-forward DNN is trained to predict the real and imaginary components of the ideal complex filter in the STFT domain, i.”
PHASEN: A Phase-and-Harmonics-Aware Speech Enhancement Network (2019, methodology): “Williamson et al. (Williamson, Wang, and Wang 2016) observed that, in Cartesian coordinates, structure exists in both real and imaginary components of the cIRM, so they designed deep neural network (DNN)-based methods to estimate the real and imaginary parts of cIRM.”
Deep Learning Based Phase Reconstruction for Speaker Separation: A Trigonometric Perspective (2018, result): “This distinguishes our approach from studies that directly predict unbounded or unconstrained phase differences [22], [23], clean phases [24] and real and imaginary components of target sources [25], [26], or fully complex neural network approaches [27].”
FRCRN: Boosting Feature Representation Using Frequency Recurrence for Monaural Speech Enhancement (2022, positive): “To further improve performance, we implement the FRCRN in complex-valued operations and predict the complex Ideal Ratio Mask (cIRM) [10].”
VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency (2021, methodology): “To perform separation, we predict complex ideal ratio masks (cIRM) [76] MA1 , MA2 , MB1 and MB2 to separate clean speech for the corresponding speakers from X1 and X2.”
Speech Enhancement and Dereverberation With Diffusion-Based Generative Models (2022, background): “We represent our data in the complex-valued STFT domain, as it has been observed that both real and imaginary parts of clean speech spectrograms exhibit clear structure and are therefore amenable to deep learning models [4].”
An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation (2020, background): “Therefore, the phase sensitive mask (PSM) [58], [261] and the complex ratio mask (CRM) [261], [271] have been proposed.”
Exploring the Best Loss Function for DNN-Based Low-latency Speech Enhancement with Temporal Convolutional Networks (2020, background): “While approaches that estimate not only the amplitude but also the phase have been proposed [7, 8], in [9], a framework that optimizes the network such that the separated signals satisfy both STFT consistency and mixture consistency was proposed for real-valued as well as complex-valued masks.”
Phase-aware Speech Enhancement with Deep Complex U-Net (2019, methodology): “Previous studies tried this joint estimation approach bounding the range of the cRM (Williamson et al., 2016; Ephrat et al., 2018).”
Learning Complex Spectral Mapping with Gated Convolutional Recurrent Networks for Monaural Speech Enhancement (2019, methodology): “For complex spectral mapping, the estimation of the real component and that of the imaginary component can be considered as two related subtasks (see [43]).”
Fullsubnet: A Full-Band and Sub-Band Fusion Model for Real-Time Single-Channel Speech Enhancement (2020, methodology): “Learning target is the spectral feature of clean speech or a certain mask (e.g., Ideal Binary Mask [8], Ideal Ratio Mask [9], complex Ideal Ratio Mask (cIRM) [10]).”
Raw waveform-based speech enhancement by fully convolutional networks (2017, methodology): “[16, 19] employed a DNN to estimate the complex ratio mask (cRM) from a set of complementary features, and then the magnitude and phase can be jointly enhanced through cRM.”
FullSubNet+: Channel Attention Fullsubnet with Complex Spectrograms for Speech Enhancement (2022, background): “As another main-stream, the frequency-domain approaches [4–7] generally take the noisy spectral feature as the input, and their learning target is the clean spectral feature or the mask (e.g., Ideal Binary Mask [8], Ideal Ratio Mask [9], complex Ideal Ratio Mask (cIRM) [10]).”
Interactive Speech and Noise Modeling for Speech Enhancement (2020, background): “They typically estimate a real or complex ratio mask for each T-F bin to map noisy spectra to speech spectra (Williamson, Wang, and Wang 2015; Wang, Narayanan, and Wang 2014; Choi et al. 2019) or directly predict the speech representation (Xu et al. 2013; Tan and Wang 2018).”