Voxceleb: Large-scale speaker verification in the wild

By Arsha Nagrani, Joon Son Chung, Weidi Xie, Andrew Zisserman. 2020

848 citations recorded by the source corpus.

Abstract The objective of this work is speaker recognition under noisy and unconstrained conditions. We make two key contributions. First, we introduce a very large-scale audio-visual dataset collected from open source media using a fully automated pipeline. Most existing datasets for speaker identification contain samples obtained under quite constrained conditions, and usually require manual annotations, hence are limited in size. We propose a pipeline based on computer vision techniques to create the dataset from open-source media. Our pipeline involves obtaining videos from YouTube; performing active speaker verification using a two-stream synchronization Convolutional Neural Network (CNN), and confirming the identity of the speaker using CNN based facial recognition. We use this pipeline to curate VoxCeleb which contains contains over a million ‘real-world’ utterances from over 6000 speakers. This is several times larger than any publicly available speaker recognition dataset. Second, we develop and compare different CNN architectures with various aggregation methods and training loss functions that can effectively recognise identities from voice under various conditions. The models trained on our dataset surpass the performance of previous works by a significant margin.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

CHiME-6 Challenge: Tackling Multispeaker Speech Recognition for Unsegmented Recordings (2020, methodology): “sh in Track 2 additionally includes SAD model training on the CHiME-6 dataset, and diarization model training on the VoxCeleb dataset [43].”
SUPERB: Speech processing Universal PERformance Benchmark (2021, methodology): “Vox-Celeb1 [26] is used without VoxCeleb2 training data and noise augmentation.”
Wespeaker: A Research and Production Oriented Speaker Embedding Learning Toolkit (2022, background): “VoxCeleb dataset [31] has been released by Oxford and become one of the most popular text-independent speaker recognition datasets.”
Expressive Neural Voice Cloning (2021, methodology): “The Resemblyzer speaker encoder [15, 14] used in our experiments is trained on the VoxCeleb [16], VoxCeleb2 [4] and LibriSpeech-other [19] datasets containing a total of 8.4k speakers.”
End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractors (2020, methodology): “We note that our system is based on 8 kHz audio, while others use 16 kHz audio with additional training data from VoxCeleb datasets [42].”
Emotional Voice Conversion: Theory, Databases and ESD (2021, neutral): “There are other large speech databases as well, such as LibriTTS [172] and VoxCeleb [173].”
An Overview of Voice Conversion and Its Challenges: From Statistical Modeling to Deep Learning (2020, methodology): “The VoxCeleb database [306] is further a larger scale speech database”
Towards Neural Diarization for Unlimited Numbers of Speakers Using Global and Local Attractors (2021, background): “Such information is accessible in the simulated mixtures created from single-speaker recordings ( e.g ., NIST SRE, Librispeech [37], VoxCeleb [38]), but is not always available in real conversation datasets ( e.g ., DIHARD [17, 3]).”
Didispeech: A Large Scale Mandarin Speech Corpus (2020, methodology): “The speaker encoder is a pre-trained model(2) for the speaker verification task using kaldi trained on VoxCeleb1&2 [28, 29], and is able to extract a 512-dimensional x-vector embedding from a reference audio.”
Masked Autoencoders that Listen (2022, background): “We perform an extensive evaluation on six tasks, including audio classiﬁcation on AudioSet (AS-2M, AS-20K) and Environmental Sound Classiﬁcation (ESC-50), and speech classiﬁcation on Speech Commands (SPC-1 and SPC-2) and VoxCeleb (SID).”
Two-branch Recurrent Network for Isolating Deepfakes in Videos (2020, background): “In the short term, we plan to measure the impact of data augmentation and the usage of additional external natural faces [43].”
Vggsound: A Large-Scale Audio-Visual Dataset (2020, result): “In contrast to these manually curated datasets, recent papers have demonstrated the possibility of collecting high-quality human speech datasets in an automated and scalable manner by using computer vision algorithms [11, 12, 13].”
SSAST: Self-Supervised Audio Spectrogram Transformer (2021, methodology): “VoxCeleb 1 (SID) We use the VoxCeleb 1 dataset (Nagrani et al. 2020) that contains speech from 1,251 speakers for the speaker identification task.”
A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speech (2023, methodology): “For human evaluation, to ensure speaker diversity, we select 40 utterances from different speakers using the test set of VoxCeleb (Na-grani et al. 2019) as the speaker reference audios, which are also noisy spontaneous speech.”
End-to-End anti-spoofing with RawNet2 (2020, methodology): “When trained using the VoxCeleb2 database, RawNet2 gives a 20% relative improvement compared to an x-vector based system [19] whereas, using the expanded VoxCeleb1-E evaluation dataset protocol, it gives a 13% relative improvement compared to a Thin-ResNet-34 based system [20].”
Spot the conversation: speaker diarisation in the wild (2020, methodology): “Inspired by the recent success of automatic audio-visual dataset creation pipelines (VoxCeleb [8, 9, 10], VGGSound [11]), we propose a scalable, audio-visual method for speaker diarisation in web videos.”