Supervised Speech Separation Based on Deep Learning: An Overview

By Deliang Wang, Jitong Chen. 2017

1,601 citations recorded by the source corpus.

Speech separation is the task of separating target speech from background interference. Traditionally, speech separation is studied as a signal processing problem. A more recent approach formulates speech separation as a supervised learning problem, where the discriminative patterns of speech, speakers, and background noise are learned from training data. Over the past decade, many supervised separation algorithms have been put forward. In particular, the recent introduction of deep learning to supervised speech separation has dramatically accelerated progress and boosted separation performance. This paper provides a comprehensive overview of the research on deep learning based supervised speech separation in the last several years. We first introduce the background of speech separation and the formulation of supervised separation. Then, we discuss three main components of supervised separation: learning machines, training targets, and acoustic features. Much of the overview is on separation algorithms where we review monaural methods, including speech enhancement (speech-nonspeech separation), speaker separation (multitalker separation), and speech dereverberation, as well as multimicrophone techniques. The important issue of generalization, unique to supervised learning, is discussed. This overview provides a historical perspective on how advances are made. In addition, we discuss a number of conceptual issues, including what constitutes the target source.

Subfields: Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Signal Processing

1. Shinji Watanabe
2. Jinyu Li
3. Deliang Wang
4. Dong Yu
5. John R. Hershey
6. Nima Mesgarani
7. Yossi Adi
8. Yi Luo

Citations in the corpus

• Exploring Speech Enhancement with Generative Adversarial Networks for Robust Speech Recognition (2017, methodology): “Besides the conventional enhancement techniques [3], deep neural networks have been widely adopted to either directly reconstruct clean speech [7, 8] or estimate masks [9, 10, 11] from the noisy signals.”
• Voice Separation with an Unknown Number of Multiple Speakers (2020, methodology): “Additionally, most previous research was conducted at the spectrum level of the input signal (Wang & Chen, 2018; Ernst et al., 2018).”
• Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation (2018, methodology): “on approaches have been formulated in the time-frequency (T-F, or spectrogram) representation of the mixture signal, which is estimated from the waveform using the short-time Fourier transform (STFT) [1]. Speech separation methods in the TF domain aim to approximate the cleaned spectrogram of the in”
• Looking to listen at the cocktail party (2018, neutral): “In previous work [Wang and Chen 2017; Wang et al. 2014], multiplicative masks have been observed to work better than alternatives such as direct prediction The visual streams take as input thumbnails of detected faces in each frame in the video, and the audio stream takes as input the video’s…”
• The Sound of Pixels (2018, background): “More recently, deep learning methods have gained popularity [22,23].”
• End-to-End Speech Separation with Unfolded Iterative Phase Reconstruction (2018, background): “Sigmoidal units are dominantly used in the output layer of deep learning based T-F masking [36, 35], partly because they can model well data with bi-modal distribution [37], such as the IRM [38] and its variants [36].”
• The Conversation: Deep Audio-Visual Speech Enhancement (2018, neutral): “by using voice characteristics of a known speaker [3, 4, 5, 6, 7].”
• A Review of Speaker Diarization: Recent Advances with Deep Learning (2021, methodology): “, [81, 82, 83], herein most of the recent techniques used in the DIHARD challenge series [84, 85, 86], LibriCSS meeting recognition task [87, 88], and CHiME-6 challenge track 2 [89, 90, 91] are covered.”
• A New Framework for CNN-Based Speech Enhancement in the Time Domain (2019, neutral): “In the last few years, supervised methods for speech enhancement using deep neural networks (DNNs) have become the mainstream [3].”
• An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation (2020, methodology): “The survey article by Wang and Chen [261] is the most extensive overview on deep-”
• Co-Separating Sounds of Visual Objects (2019, background): “We find spectrogram masks to work better than direct prediction of spectrograms or raw waveforms for source separation, confirming reports in [49, 8, 13].”
• MetricGAN+: An Improved Version of MetricGAN for Speech Enhancement (2021, methodology): “Most mask-based speech enhancement methods [24] apply a sigmoid activation to the output layer to constrain the mask to be between 0 and 1.”
• TCNN: Temporal Convolutional Neural Network for Real-time Speech Enhancement in the Time Domain (2019, neutral): “In the past few years, deep learning based supervised methods have become the mainstream for speech enhancement [5].”
• Fréchet Audio Distance: A Metric for Evaluating Music Enhancement Algorithms (2018, neutral): “Existing research has looked into techniques for speech separation [23] and speech enhancement [12] as well as separating music into its instrumental components [13] or removing the vocals to produce a karaoke version of the track [9].”
• Deep Learning for Audio Signal Processing (2019, methodology): “Besides conventional enhancement techniques [129], deep neural networks have been widely adopted to either directly reconstruct clean speech [132], [133] or estimate masks [134]–[136] from”