Continuous Speech Separation: Dataset and Analysis

By Zhuo Chen, Takuya Yoshioka, Liang Lu, Tianyan Zhou, Zhong Meng, Yi Luo, Jian Wu, Jinyu Li. 2020

256 citations recorded by the source corpus.

This paper describes a dataset and protocols for evaluating continuous speech separation algorithms. Most prior speech separation studies use pre-segmented audio signals, which are typically generated by mixing speech utterances on computers so that they fully overlap. Also, the separation algorithms have often been evaluated based on signal-based metrics such as signal-to-distortion ratio. However, in natural conversations, speech signals are continuous and contain both overlapped and overlap-free regions. In addition, the signal-based metrics only have weak correlation with automatic speech recognition (ASR) accuracy. Not only does this make it hard to assess the practical relevance of the tested algorithms, it also hinders researchers from developing systems that can be readily applied to real scenarios. In this paper, we define continuous speech separation (CSS) as a task of generating a set of non-overlapped speech signals from a continuous audio stream that contains multiple utterances that are partially overlapped by a varying degree. A new real recording dataset, called LibriCSS, is derived from LibriSpeech by concatenating the corpus utterances to simulate conversations and capturing the audio replays with far-field microphones. A Kaldi-based ASR evaluation protocol is established by using a well-trained multi-conditional acoustic model. A recently proposed speaker-independent CSS algorithm is investigated by using LibriCSS. The dataset and evaluation scripts are made available to facilitate the research in this direction1.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Shinji Watanabe
3. Zhuo Chen
4. Shujie Liu
5. Yu Wu
6. Dong Yu
7. Takuya Yoshioka
8. Sanyuan Chen

In Signal Processing

1. Jinyu Li
2. Shinji Watanabe
3. Zhuo Chen
4. Takuya Yoshioka
5. Dong Yu
6. Shujie Liu
7. Yu Wu
8. John R. Hershey

Citations in the corpus

• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing (2021, methodology): “WavLM achieves SOTA performance on the speech separation LibriCSS benchmark [20], and significantly outperforms the previous Conformer model [21] by a 27.”
• Ultra Fast Speech Separation Model with Teacher Student Learning (2021, methodology): “We conducted both the utterance-wise evaluation and continuous input evaluation (refer to [10] for the two evaluation schemes).”
• Investigation of Practical Aspects of Single Channel Speech Separation for ASR (2021, methodology): “We evaluate our model in both utterance-wise and continuous evaluation as defined in [16].”
• A Review of Speaker Diarization: Recent Advances with Deep Learning (2021, methodology): “The end-to-end SA-ASR model was evaluated by using the LibriCSS dataset [81], and exhibited significantly better cpWER than the combination of multitalker ASR and speaker diarization [199].”
• Encoder-Decoder Based Attractor Calculation for End-to-End Neural Diarization (2021, background): “It can be placed as a downstream task of automatic speech recognition (ASR), in which speaker information is tagged to each transcribed utterance [1]–[3].”
• Large-Scale Pre-Training of End-to-End Multi-Talker ASR for Meeting Transcription with Single Distant Microphone (2021, methodology): “Another approach is to apply a speech separation sys-tem, followed by the ASR system (e.g., [14, 15]).”
• LibriMix: An Open-Source Dataset for Generalizable Speech Separation (2020, background): “LibriCSS [17], an open-source dataset for sparsely overlapping continuous speech separation, has recently been released.”
• Towards Neural Diarization for Unlimited Numbers of Speakers Using Global and Local Attractors (2021, background): “To extend EEND-EDA to an unlimited number of speakers, we focus on the fact that the number of speakers who speak in a short period is suf-ficiently small [15, 16].”
• AISHELL-4: An Open Source Dataset for Speech Enhancement, Separation, Recognition and Speaker Diarization in Conference Scenario (2021, background): “Therefore, most of the previous works on meeting transcription or relevant tasks such as overlapped speech recognition employ synthetic datasets for experimentation [4, 5, 6].”