Speaker Diarization with LSTM

By Quan Wang, Carlton Downey, Li Wan, Philip Andrew Mansfield, Ignacio Lopez-Moreno. IEEE International Conference on Acoustics, Speech, and Signal Processing, 2017

344 citations recorded by the source corpus.

For many years, i-vector based audio embedding techniques were the dominant approach for speaker verification and speaker diarization applications. However, mirroring the rise of deep learning in various domains, neural network based audio embeddings, also known as d-vectors, have consistently demonstrated superior speaker verification performance. In this paper, we build on the success of d-vector based speaker verification systems to develop a new d-vector based approach to speaker diarization. Specifically, we combine LSTM-based d-vector audio embeddings with recent work in nonparametric clustering to obtain a state-of-the-art speaker diarization system. Our system is evaluated on three standard public datasets, suggesting that d-vector based diarization systems offer significant advantages over traditional i-vector based systems. We achieved a 12.0% diarization error rate on NIST SRE 2000 CALLHOME, while our model is trained with out-of-domain data from voice search logs.

Subfields: Artificial Intelligence, Signal Processing

View Paper: https://www.semanticscholar.org/paper/507271f26a1c40bb128d8e09289e2054d304e49c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

WhisperX: Time-Accurate Speech Transcription of Long-Form Audio (2023, background): “This has catastrophic consequences for buffered transcription of long-form and other timestamp-sensitive tasks such as speaker diarization [13, 14], lip-reading [15] and audiovisual learning [16].”
End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractors (2020, background): “Progress on better speaker embeddings, such as x-vectors [10, 11] and d-vectors [12, 13], have enabled accurate clusteringbased diarization.”
Speaker Recognition for Multi-speaker Conversations Using X-vectors (2019, methodology): “Mirroring progress in speaker recognition, recent systems have replaced i-vectors with DNN-based embeddings for capturing speaker characteristics [13, 14, 15].”
Speaker recognition based on deep learning: An overview (2020, methodology): “Popular similarity measurements include cosine similarity [169] and PLDAbases similarity [167, 179].”
VoiceFilter: Targeted Voice Separation by Speaker-Conditioned Spectrogram Masking (2018, unknown): “Speaker A”
Neural Speaker Diarization with Speaker-Wise Chain Rule (2020, methodology): “Clustering methods commonly used for speaker diarization are agglomerative hierarchical clustering (AHC) [15, 18, 19], k-means clustering [17, 20], and spectral clustering [20].”
A Review of Speaker Diarization: Recent Advances with Deep Learning (2021, methodology): “However, the k-means algorithm generally underperforms [39, 41] the well-known clustering algorithms such as spectral clustering and AHC.”
The Second DIHARD Diarization Challenge: Dataset, task, and baselines (2019, background): “These error rates rates are more than double the state-of-the-art for CALLHOME [16] at the time [4, 5].”
Encoder-Decoder Based Attractor Calculation for End-to-End Neural Diarization (2021, methodology): “Eigen value analysis on the graph Laplacian of a similarity matrix calculated from frame-wise embeddings is one way to estimate the number of speakers explicitly [6], [7].”
Towards Neural Diarization for Unlimited Numbers of Speakers Using Global and Local Attractors (2021, methodology): “For example, Gaussian blur was applied to smooth the afﬁnity matrix calculated from d-vectors extracted using sliding window [23].”
End-to-End Neural Speaker Diarization with Permutation-Free Objectives (2019, methodology): “The system first extracts speaker representations such as i-vectors [12, 13, 17, 19], d-vectors [18, 20], or x-vectors [16, 21].”
Attention-Based Models for Text-Dependent Speaker Verification (2017, background): “The same attention mechanisms, especially the ones using shared-parameter scoring functions, could potentially be used to improve text-independent speaker verification models [16] and speaker diarization systems [17].”