Interactive Speech and Noise Modeling for Speech Enhancement

By Chengyu Zheng, Xiulian Peng, Yuan Zhang, Sriram Srinivasan, Yan Lu. 2020

114 citations recorded by the source corpus.

Speech enhancement is challenging because of the diversity of background noise types. Most of the existing methods are focused on modelling the speech rather than the noise. In this paper, we propose a novel idea to model speech and noise simultaneously in a two-branch convolutional neural network, namely SN-Net. In SN-Net, the two branches predict speech and noise, respectively. Instead of information fusion only at the final output layer, interaction modules are introduced at several intermediate feature domains between the two branches to benefit each other. Such an interaction can leverage features learned from one branch to counteract the undesired part and restore the missing component of the other and thus enhance their discrimination capabilities. We also design a feature extraction module, namely residual-convolution-and-attention (RA), to capture the correlations along temporal and frequency dimensions for both the speech and the noises. Evaluations on public datasets show that the interaction module plays a key role in simultaneous modeling and the SN-Net outperforms the state-of-the-art by a large margin on various evaluation metrics. The proposed SN-Net also shows superior performance for speaker separation.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xue Jiang
2. Jordi Pons
3. Xiulian Peng
4. Yan Lu
5. Santiago Pascual
6. Joan Serrà
7. Huaying Xue
8. Xuyi Zhuang

In Signal Processing

1. Jordi Pons
2. Xue Jiang
3. Santiago Pascual
4. Joan Serrà
5. Recep Oguz Araz
6. Davide Scaini
7. Xiulian Peng
8. Yan Lu

Citations in the corpus

• End-to-End Neural Speech Coding for Real-Time Communications (2022, positive): “We employ causal gated blocks in decoder and output an amplitude gain and the phase for reconstruction, similar to that in [17].”
• Universal Speech Enhancement with Score-based Diffusion (2022, background): “The state of the art for speech denoising and dereverberation is dominated by regression and adversarial approaches (Défossez et al., 2020; Fu et al., 2021; Su et al., 2021; Isik et al., 2020; Hao et al., 2021; Kim & Seo, 2021; Zheng et al., 2021; Kataria et al., 2021).”
• FB-MSTCN: A Full-Band Single-Channel Speech Enhancement Method Based on Multi-Scale Temporal Convolutional Network (2022, methodology): “For example, in wide-band real-time scenarios, most methods [4, 5, 6, 7] have the input frame size set to 20 ms or 32 ms, corresponding to 320 and 512 points of Fourier transform analysis, respectively.”