End-to-End Neural Speaker Diarization with Permutation-Free Objectives

By Yusuke Fujita, Naoyuki Kanda, Shota Horiguchi, Kenji Nagamatsu, Shinji Watanabe. 2019

312 citations recorded by the source corpus.

In this paper, we propose a novel end-to-end neural-network-based speaker diarization method. Unlike most existing methods, our proposed method does not have separate modules for extraction and clustering of speaker representations. Instead, our model has a single neural network that directly outputs speaker diarization results. To realize such a model, we formulate the speaker diarization problem as a multi-label classification problem, and introduces a permutation-free objective function to directly minimize diarization errors without being suffered from the speaker-label permutation problem. Besides its end-to-end simplicity, the proposed method also benefits from being able to explicitly handle overlapping speech during training and inference. Because of the benefit, our model can be easily trained/adapted with real-recorded multi-speaker conversations just by feeding the corresponding multi-speaker segment labels. We evaluated the proposed method on simulated speech mixtures. The proposed method achieved diarization error rate of 12.28%, while a conventional clustering-based system produced diarization error rate of 28.77%. Furthermore, the domain adaptation with real-recorded speech provided 25.6% relative improvement on the CALLHOME dataset. Our source code is available online at this https URL.

Subfields: Artificial Intelligence, Natural Language Processing, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Speech & Audio

In Signal Processing

Citations in the corpus

Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation (2021, background): “Example applications include speaker diarization (Fujita et al., 2019), vector graphics generation (Carlier et al., 2020), object detection (Carion et al., 2020) and point cloud generation (Achlioptas et al., 2018).”
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing (2021, methodology): “Based on the simulation pipeline introduced in [66], we generate almost 7000 hours simulation data by setting Nspk = 3 and β = 10.”
SUPERB: Speech processing Universal PERformance Benchmark (2021, methodology): “We employ the end-to-end training scheme with permutation-invariant training (PIT) loss [42] to SD , instead of using clustering-based methods.”
Powerset multi-class cross entropy loss for neural speaker diarization (2023, background): “To circumvent these limitations, a new family of approaches have recently emerged, rethinking speaker diarization completely [3, 4, 5].”
Advances in integration of end-to-end neural and clustering-based diarization for real conversational speech (2021, methodology): “For the training, we used simulated mixtures created from Switchboard-2 (Phase I & II & III), Switchboard Cellular (Part 1 & 2), and the NIST Speaker Recognition Evaluation (2004 & 2005 & 2006 & 2008) for speech, and the MUSAN corpus [23] for noise with simulated room impulse responses used in [24],”
End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractors (2020, methodology): “First, we evaluated our method on the 2-speaker condition like the one in [15, 16].”
Neural Speaker Diarization with Speaker-Wise Chain Rule (2020, methodology): “End-to-end neural diarization (EEND) [27, 28] is one of such clustering-free methods.”
Encoder-Decoder Based Attractor Calculation for End-to-End Neural Diarization (2021, methodology): “End-to-end neural diarization (EEND) [11], [12] is a method for estimating multiple speakers’ speech activities simultaneously from an input recording.”
End-to-end speaker segmentation for overlap-aware resegmentation (2021, methodology): “This model is directly inspired by recent advances in end-to-end speaker diarization and, in particular, the growing End-to-End Neural Diarization (EEND) family of approaches developed by Hitachi [5, 6, 7].”
A Review of Speaker Diarization: Recent Advances with Deep Learning (2021, methodology): “Speaker clustering constrained by the estimated speech activity by EEND is applied to align the estimated speakers among the different processing blocks.”