Multimodal SpeakerBeam: Single Channel Target Speech Extraction with Audio-Visual Speaker Clues

By Tsubasa Ochiai, Marc Delcroix, Keisuke Kinoshita, Atsunori Ogawa, Tomohiro Nakatani. 2019

70 citations recorded by the source corpus.

Recently, with the advent of deep learning, there has been significant progress in the processing of speech mixtures. In particular, the use of neural networks has enabled target speech extraction, which extracts speech signal of a target speaker from a speech mixture by utilizing auxiliary clue representing the characteristics of the target speaker. For example, audio clues derived from an auxiliary utterance spoken by the target speaker have been used to characterize the target speaker. Audio clues should capture the fine-grained characteristic of the target speaker’s voice (e.g., pitch). Alternatively, visual clues derived from a video of the target speaker’s face speaking in the mixture have also been investigated. Visual clues should mainly capture the phonetic information derived from lip movements. In this paper, we propose a novel target speech extraction scheme that combines audio and visual clues about the target speaker to take advantage of the information provided by both modalities. We introduce an attention mechanism that emphasizes the most informative speaker clue at every time frame. Experiments on mixture of two speakers demonstrated that our proposed method using audio-visual speaker clues significantly improved the extraction performance compared with the conventional methods using either audio or visual speaker clues.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

SEANet: A Multi-modal Speech Enhancement Network (2020, methodology): “A similar approach is presented in [20], which adopts an attention mechanism to weight the contribution of the audio and visual modalities.”
Neural Target Speech Extraction: An overview (2023, methodology): “Many works have combined audio and visual clues [4], [33], and some have even added spatial clues [36].”
An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation (2020, background): “SDR [252] 2006 Specifically designed for blind audio [7], [10], [17], [42], [55], [65], [85] source separation [107]–[109], [136], [153], [154], [169] [164], [165], [183], [192], [195], [203] [208], [220]–[222]”