Vggsound: A Large-Scale Audio-Visual Dataset

By Honglie Chen, Weidi Xie, Andrea Vedaldi, Andrew Zisserman. 2020

917 citations recorded by the source corpus.

Our goal is to collect a large-scale audio-visual dataset with low label noise from videos ‘in the wild’ using computer vision techniques. The resulting dataset can be used for training and evaluating audio recognition models. We make three contributions. First, we propose a scalable pipeline based on computer vision techniques to create an audio dataset from open-source media. Our pipeline involves obtaining videos from YouTube; using image classification algorithms to localize audio-visual correspondence; and filtering out ambient noise using audio verification. Second, we use this pipeline to curate the VGGSound dataset consisting of more than 200k videos for 300 audio classes. Third, we investigate various Convolutional Neural Network (CNN) architectures and aggregation approaches to establish audio recognition baselines for our new dataset. Compared to existing audio datasets, VGGSound ensures audio-visual correspondence and is collected under unconstrained conditions. Code and the dataset are available at http://www.robots.ox.ac.uk/~vgg/data/vggsound/.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Evaluation & Benchmarks

In Speech & Audio

In Signal Processing

Citations in the corpus

FSD50K: An Open Dataset of Human-Labeled Sound Events (2020, background): “Lastly, a recent large-vocabulary dataset with a substantial amount of data is VGGSound [41], an audio-visual dataset consisting of ≈ 200k video clips from YouTube encompassing 300 classes.”
AudioGen: Textually Guided Audio Generation (2022, background): “We use a set of several datasets: AudioSet (Gemmeke et al., 2017), BBC sound effects 1 , AudioCaps (Kim et al., 2019), Clotho v2 (Drossos et al., 2020), VGG-Sound (Chen et al., 2020), FSD50K (Fonseca et al., 2021), Free To Use Sounds 2 , Sonniss Game Effects 3 , WeSoundEffects 4 , Paramount Motion -”
Diffsound: Discrete Diffusion Model for Text-to-Sound Generation (2022, background): “In contrast, Iashin and Rahtu [3] trains their model using VGGSound dataset [9], which has over 200 K audio-video pairs.”
Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models (2023, methodology): “We utilize HiFi-GAN (Kong et al., 2020) (V1) trained on VGGSound dataset (Chen et al., 2020a) as the vocoder to synthesize waveform from the generated mel-spectrogram in all our experiments.”
Wav2CLIP: Learning Robust Audio Representations from Clip (2021, positive): “VGGSound [22] is a very large audio-visual dataset with the most and diverse general sound classes among all.”
What Makes Multimodal Learning Better than Single (Provably) (2021, background): “Inspired from the success of deep learning [26, 21], deep multi-modal research is also activated, which covers fields like audio-visual learning [8, 43], RGB-D semantic segmentation [35, 20] and Visual Question Answering [17, 2].”
Perception Test: A Diagnostic Benchmark for Multimodal Video Models (2023, background): “…on one of the following data sources: (i) Videos collected from the web or repositories like Youtube, e.g. Kinetics [36], ActivityNet [9], VGGSound [11], HVU [18], ActivityNet-QA [56], tGIFQA [33]; (ii) Videos collected on demand, filmed by volunteers doing arbitrary activities in indoor or…”
Attention Bottlenecks for Multimodal Fusion (2021, methodology): “VGGSound [12] contains almost 200K video clips of length 10s, annotated with 309 sound classes consisting of human actions, sound-emitting objects and human-object interactions.”
Computational bioacoustics with deep learning: a review and roadmap (2021, methodology): “This is indicative only, since not all articles clearly state whether an off-the-shelf model is used, some articles use modiﬁed/derived versions, and some use multiple architectures. et al., 2021; Xie et al., 2020; Tzirakis et al., 2020; Li et al., 2019).”
ImageBind One Embedding Space to Bind Them All (2023, background): “50 Acc 400 Clotho (Clotho) [16] Retrieval - Recall 1045 AudioCaps (AudioCaps) [36] Retrieval - Recall 796 VGGSound (VGGS) [8] Audio cls.”
Self-Supervised Remote Sensing Feature Learning: Learning Paradigms, Challenges, and Future Works (2022, background): “the attention of scholars [218], [219], [243], [244].”
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action (2023, neutral): “S ound [ 24 ] A ud i o C a p s [ 93 ] K i n e ti c s -”
Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation (2022, methodology): “We show that LAION-Audio-630, AudioSet with keyword-to-caption augmentation, and feature fusion effectively leads to better audio understanding, task performance, and enables effective learnings on variable-length data.”
Dataset Condensation with Differentiable Siamese Augmentation (2021, methodology): “An orthogonal direction to increase data efficiency and thus generalization performance is data augmentation, a technique to expand training set with semantic-preserving transformations (Krizhevsky et al., 2012; Zhang et al., 2018; Yun et al., 2019; Chen et al., 2020b; Chen & He, 2020).”
Audio Retrieval with Natural Language Queries (2021, methodology): “We use a subset of the data, excluding a small number of samples for which either: (i) the YouTube-hosted source video is no longer available, (ii) the source video overlaps with the training partition of the VGGSound dataset [44].”
Poisoning Web-Scale Training Datasets is Practical (2023, background): “Critically, the maintainer does not provide a snapshot of the data associated with urli, due either to untenable storage costs [3,4,17,31], privacy concerns [13, 72], or copyright limitations [16].”
Multi-modal Self-Supervision from Generalized Data Transformations (2020, methodology): “Supervised [17] 51.”