AVA-Speech: A Densely Labeled Dataset of Speech Activity in Movies

By Sourish Chaudhuri, Joseph Roth, D. Ellis, Andrew C. Gallagher, Liat Kaver, Radhika Marvin, Caroline Pantofaru, Nathan Reale, Loretta Guarino Reid, Kevin Wilson and 1 more. 2018

54 citations recorded by the source corpus.

Speech activity detection (or endpointing) is an important processing step for applications such as speech recognition, language identification and speaker diarization. Both audio- and vision-based approaches have been used for this task in various settings, often tailored toward end applications. However, much of the prior work reports results in synthetic settings, on task-specific datasets, or on datasets that are not openly available. This makes it difficult to compare approaches and understand their strengths and weaknesses. In this paper, we describe a new dataset which we will release publicly containing densely labeled speech activity in YouTube videos, with the goal of creating a shared, available dataset for this task. The labels in the dataset annotate three different speech activity conditions: clean speech, speech co-occurring with music, and speech co-occurring with noise, which enable analysis of model performance in more challenging conditions based on the presence of overlapping noise. We report benchmark performance numbers on AVA-Speech using off-the-shelf, state-of-the-art audio and vision models that serve as a baseline to facilitate future research.

Subfields: Cognitive Neuroscience, Evaluation & Benchmarks, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Evaluation & Benchmarks

In Speech & Audio

In Signal Processing

Citations in the corpus

Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “Datasets like VoxCeleb [39], AVA Speech [31], AVA active speaker [192], AVDIAR [83], and EasyCom [53] support this research.”
Ava Active Speaker: An Audio-Visual Dataset for Active Speaker Detection (2019, methodology): “Previous annotations released on the videos in the AVA dataset have contributed action recognition (associated with a bounding box entity) [24] and audio speech activity labels (not associated with speciﬁc entities) [8].”