SpeakerBeam: Speaker Aware Neural Network for Target Speaker Extraction in Speech Mixtures

By Kateřina Žmolíková, Marc Delcroix, Keisuke Kinoshita, Tsubasa Ochiai, Tomohiro Nakatani, Lukáš Burget, Jan Honza Cernocký. 2019

288 citations recorded by the source corpus.

The processing of speech corrupted by interfering overlapping speakers is one of the challenging problems with regards to today's automatic speech recognition systems. Recently, approaches based on deep learning have made great progress toward solving this problem. Most of these approaches tackle the problem as speech separation, i.e., they blindly recover all the speakers from the mixture. In some scenarios, such as smart personal devices, we may however be interested in recovering one target speaker from a mixture. In this paper, we introduce SpeakerBeam, a method for extracting a target speaker from the mixture based on an adaptation utterance spoken by the target speaker. Formulating the problem as speaker extraction avoids certain issues such as label permutation and the need to determine the number of speakers in the mixture. With SpeakerBeam, we jointly learn to extract a representation from the adaptation utterance characterizing the target speaker and to use this representation to extract the speaker. We explore several ways to do this, mostly inspired by speaker adaptation in acoustic models for automatic speech recognition. We evaluate the performance on the widely used WSJ0-2mix and WSJ0-3mix datasets, and these datasets modified with more noise or more realistic overlapping patterns. We further analyze the learned behavior by exploring the speaker representations and assessing the effect of the length of the adaptation data. The results show the benefit of including speaker information in the processing and the effectiveness of the proposed method.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Zhuo Chen
3. Dong Yu
4. Takuya Yoshioka
5. Shujie Liu
6. Shinji Watanabe
7. Naoyuki Kanda
8. Marc Delcroix

In Signal Processing

1. Jinyu Li
2. Zhuo Chen
3. Dong Yu
4. Shinji Watanabe
5. Takuya Yoshioka
6. Marc Delcroix
7. Naoyuki Kanda
8. Shujie Liu

Citations in the corpus

• SpeechX: Neural Codec Language Model as a Versatile Speech Transformer (2023, background): “In the context of transforming nonclean speech, another particularly useful task is target speaker extraction [22], [23], [24].”
• Neural Target Speech Extraction: An overview (2023, methodology): “For example, using a beamformer usually extracts a higher-quality signal due to employing a spatial linear filter to perform extraction, which can benefit ASR applications [10].”