End-to-end speaker segmentation for overlap-aware resegmentation

By Hervé Bredin, Antoine Laurent. 2021

218 citations recorded by the source corpus.

Speaker segmentation consists in partitioning a conversation between one or more speakers into speaker turns. Usually addressed as the late combination of three sub-tasks (voice activity detection, speaker change detection, and overlapped speech detection), we propose to train an end-to-end segmentation model that does it directly. Inspired by the original end-to-end neural speaker diarization approach (EEND), the task is modeled as a multi-label classification problem using permutation-invariant training. The main difference is that our model operates on short audio chunks (5 seconds) but at a much higher temporal resolution (every 16ms). Experiments on multiple speaker diarization datasets conclude that our model can be used with great success on both voice activity detection and overlapped speech detection. Our proposed model can also be used as a post-processing step, to detect and correctly assign overlapped speech regions. Relative diarization error rate improvement over the best considered baseline (VBx) reaches 17% on AMI, 13% on DIHARD 3, and 13% on VoxConverse.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, methodology): “Since the speech in GigaSpeech and WenetSpeech does not have speaker identities and multiple speakers may appear in a speech clip, we process the datasets with an open-source automatic speaker diarization model 6 [6, 5].”
Intelligent Clinical Documentation: Harnessing Generative AI for Patient-Centric Clinical Note Generation (2024, methodology): “[11] This model is an extension of the original Whisper model, specifically designed to perform fast transcription of long audio and comes in with built in support for plugging in diarization model such as pyannotate/speaker-diarization [12].”
Pyannote.audio 2.1 Speaker Diarization Pipeline: Principle, Benchmark, and Recipe (2023, methodology): “The first step consists in applying the end-to-end neural speaker segmentation model introduced in [3] using a sliding window of 5s with a step of 500ms.”
Mega-TTS 2: Zero-Shot Text-to-Speech with Arbitrary Length Speech Prompts (2023, methodology): “Besides, since the speech in GigaSpeech and WenetSpeech does not have speaker identities and multiple speakers may appear in a speech clip, we process the datasets with an open-source automatic speaker diarization model 3 (Bredin et al., 2020; Bredin & Laurent, 2021).”
Powerset multi-class cross entropy loss for neural speaker diarization (2023, methodology): “To circumvent these limitations, a new family of approaches have recently emerged, rethinking speaker diarization completely [3, 4, 5].”
Advancing Alzheimer Disease Prediction With Large Language Model–Based Linguistic Feature Analysis: Development and Validation Study (2026, unknown): “We observe that the speaker diarization method using PyAnnote [49] was prone to failure or speaker misidentification in low-volume recordings and cases where examiner and participant voices were acoustically similar, resulting in incomplete or contaminated transcripts.”