Ava Active Speaker: An Audio-Visual Dataset for Active Speaker Detection

By Joseph Roth, Sourish Chaudhuri, Ondrej Klejch, Radhika Marvin, Andrew C. Gallagher, Liat Kaver, S. Ramaswamy, Arkadiusz Stopczynski, Cordelia Schmid, Zhonghua Xi and 1 more. 2019

179 citations recorded by the source corpus.

Active speaker detection is an important component in video analysis algorithms for applications such as speaker diarization, video re-targeting for meetings, speech enhancement, and human-robot interaction. The absence of a large, carefully labeled audio-visual active speaker dataset has limited evaluation in terms of data diversity, environments, and accuracy. In this paper, we present the AVA Active Speaker detection dataset (AVA-ActiveSpeaker) which has been publicly released to facilitate algorithm development and comparison. It contains temporally labeled face tracks in videos, where each face instance is labeled as speaking or not, and whether the speech is audible. The dataset contains about 3.65 million human labeled frames spanning 38.5 hours. We also introduce a state-of-the-art, jointly trained audio-visual model for real-time active speaker detection and compare several variants. The evaluation clearly demonstrates a significant gain due to audio-visual modeling and temporal integration over multiple frames.

Subfields: Artificial Intelligence, Multimodal Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Multimodal Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “The TTM task is related to audio-visual speaker detection [7,193] and meeting understanding [21, 132, 154].”
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, background): “uTube 1806 1 650k 10:00:0 700 a Top-1/5 Err. Multi-Moments in Time [43] 2019 YouTube 850 1 1M 3:00:0 313 ML-a mAP Something-Something [44] 2018 Scripted 234 1 220k 3:81:1 174 a, n, c Top-1/5 Acc. AVA [33] 2018 Film 108 1380 410k 2:73:5 80 ML-a mAP HACS [45] 2017 YouTube 861 2 1.5M 2 :00 200 a mAP Ch”
MovieNet: A Holistic Dataset for Movie Understanding (2020, methodology): “We use tools from [81] that apply NaverNet [14] pretrained on AVA-ActiveSpeaker dataset [58] to separate speech and background sound, and stft [70] to get their features respectively in a shot with 16K Hz sampling rate and 512 windowed signal length, and concatenate them to obtain audio features.”
Deep Audio-visual Learning: A Survey (2020, background): “Classes Length* Year GRID [119] Lab 34 33,000 2006 Lombard Grid [120] Lab 54 54,000 2018 TCD TIMIT [121] Lab 62 2015 Vid TIMIT [122] Lab 43 2009 RAVDESS [123] Lab 24 2018 SEWA [124] Lab 180 2017 Speech OuluVS [125] Lab 20 1000 2009 OuluVS2 [126] Lab 52 3640 2016 Voxceleb [127] Wild 1,251 154,516 201”
An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation (2020, background): “The trend of collecting larger datasets has continued in subsequent collections which consist of materials from British television programs [8], [39], [41], generic YouTube videos [38], [185], TED talks [9], [55], movies [217] or lectures [55], [244].”
Is Someone Speaking?: Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection (2021, unknown): “The AVA-ActiveSpeaker dataset1 is derived from Hollywood movies [35].”
EgoCom: A Multi-Person Multi-Modal Egocentric Communications Dataset (2020, methodology): “Multi-modal AI datasets include AVA-ActiveSpeaker, an audio-visual dataset for speaker detection [25], VGG lip reading dataset, an audio-visual dataset for speech recognition and separation [26], Mosi, a multimodal corpus of sentiment intensity [27],”