An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation

By Daniel Michelsanti, Zheng-Hua Tan, Shi-Xiong Zhang, Yong Xu, Meng Yu, Dong Yu, Jesper Højvang Jensen. IEEE/ACM Transactions on Audio Speech and Language Processing, 2020

337 citations recorded by the source corpus.

<italic>Speech enhancement</italic> and <italic>speech separation</italic> are two related tasks, whose purpose is to extract either one or more target speech signals, respectively, from a mixture of sounds generated by several sources. Traditionally, these tasks have been tackled using signal processing and machine learning techniques applied to the available acoustic signals. Since the visual aspect of speech is essentially unaffected by the acoustic environment, <italic>visual information</italic> from the target speakers, such as lip movements and facial expressions, has also been used for speech enhancement and speech separation systems. In order to efficiently fuse acoustic and visual information, researchers have exploited the flexibility of data-driven approaches, specifically <italic>deep learning</italic>, achieving strong performance. The ceaseless proposal of a large number of techniques to extract features and fuse multimodal information has highlighted the need for an overview that comprehensively describes and discusses audio-visual speech enhancement and separation based on deep learning. In this paper, we provide a systematic survey of this research topic, focusing on the main elements that characterise the systems in the literature: <italic>acoustic features</italic>; <italic>visual features</italic>; <italic>deep learning methods</italic>; <italic>fusion techniques</italic>; <italic>training targets</italic> and <italic>objective functions</italic>. In addition, we review deep-learning-based methods for <italic>speech reconstruction from silent videos</italic> and <italic>audio-visual sound source separation for non-speech signals</italic>, since these methods can be more or less directly applied to audio-visual speech enhancement and separation. Finally, we survey commonly employed <italic>audio-visual speech datasets</italic>, given their central role in the development of data-driven approaches, and <italic>evaluation methods</italic>, because they are generally used to compare different systems and determine their performance.

Subfields: Computational Mechanics, Signal Processing

View Paper: https://www.semanticscholar.org/paper/e1859067487893f6580e934e9ee3408a2fa8b7e1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Signal Processing

Citations in the corpus

Neural Target Speech Extraction: An overview (2023, methodology): “When targeting applications for human listeners, the problem can be partly addressed using other metrics for training and evaluation that correlate better with human perception, such as short-time objective intelligibility and perceptual evaluation of speech quality [6].”