Spot the conversation: speaker diarisation in the wild

By Joon Son Chung, Jaesung Huh, Arsha Nagrani, Triantafyllos Afouras, Andrew Zisserman. 2020

198 citations recorded by the source corpus.

The goal of this paper is speaker diarisation of videos collected 'in the wild'. We make three key contributions. First, we propose an automatic audio-visual diarisation method for YouTube videos. Our method consists of active speaker detection using audio-visual methods and speaker verification using self-enrolled speaker models. Second, we integrate our method into a semi-automatic dataset creation pipeline which significantly reduces the number of hours required to annotate videos with diarisation labels. Finally, we use this pipeline to create a large-scale diarisation dataset called VoxConverse, collected from 'in the wild' videos, which we will release publicly to the research community. Our dataset consists of overlapping speech, a large and diverse speaker pool, and challenging background conditions.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency (2021, background): “Leveraging this link, cross-modal learning methods explore a range of interesting tasks: face reconstruction from audio [54], talking face generation [82], emotion recognition [5], speaker diarization [13], speech recognition [16], and speaker identification [50, 49, 39, 74, 17].”
Is Someone Speaking?: Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection (2021, unknown): “This task is an essential frontend for a wide range of multi-modal applications such as audio-visual speech recognition [1], speech separation [29], speaker diarization [14, 15] and speaker tracking [33, 34].”
Pyannote.audio 2.1 Speaker Diarization Pipeline: Principle, Benchmark, and Recipe (2023, background): “• A growing number of freely available datasets such as AISHELL-4 [11], Albayzin/RTVE [10], AliMeeting [12], AMI [13], VoxConverse [9], Ego4D [14], or This American Life [15].”
Powerset multi-class cross entropy loss for neural speaker diarization (2023, methodology): “We report performance on the official test sets of 9 different datasets: AISHELL-4 [12], AliMeeting [13], AMI [14] (with two variants), DIHARD III [21], Ego4D 2 [15], MSDWild [16], REPERE [17], and VoxConverse [22].”
End-to-end speaker segmentation for overlap-aware resegmentation (2021, methodology): “This is consistent with DIHARD3 evaluation plan [10] and AMI Full evaluation setup [15], but not with VoxConverse challenge rules that uses a 250ms collar [11].”
A Review of Speaker Diarization: Recent Advances with Deep Learning (2021, methodology): “diarization accuracy for speaker diarization on YouTube video [192].”
Wespeaker: A Research and Production Oriented Speaker Embedding Learning Toolkit (2022, methodology): “VoxConverse dataset [33] was released for the diarization track in VoxSRC 2020, which is a “in the wild” dataset collected from the Youtube Videos.”