Audio-Visual Scene Analysis with Self-Supervised Multisensory Features

By Andrew Owens, Alexei A. Efros. 2018

825 citations recorded by the source corpus.

The thud of a bouncing ball, the onset of speech as lips open -- when visual and audio events occur together, it suggests that there might be a common, underlying event that produced both signals. In this paper, we argue that the visual and audio components of a video signal should be modeled jointly using a fused multisensory representation. We propose to learn such a representation in a self-supervised way, by training a neural network to predict whether video frames and audio are temporally aligned. We use this learned representation for three applications: (a) sound source localization, i.e. visualizing the source of sound in a video; (b) audio-visual action recognition; and (c) on/off-screen audio source separation, e.g. removing the off-screen translator's voice from a foreign official's speech. Code, models, and video results are available on our webpage: this http URL

Subfields: Cognitive Neuroscience, Multimodal Learning, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Multimodal Learning

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

S4L: Self-Supervised Semi-Supervised Learning (2019, background): “Self-supervised techniques have a variety of applications in a broad range of computer vision topics [14, 35, 7, 29, 34].”
Learning Video Representations from Textual Web Supervision (2020, background): “Prior works have leveraged ambient sound [52, 5, 4, 71, 34, 51, 55], dialogue [49], and narration [70, 2, 72, 73, 46, 45, 3], all of which of which serve as useful signals.”
Self-Supervised Learning of Pretext-Invariant Representations (2019, background): “For video data, these pretext tasks learn representations by ordering video frames [1, 18, 34, 41, 51, 79, 83], tracking [62, 77], or using cross-modal signals like audio [2, 3, 19, 36, 60, 61].”
Revisiting Self-Supervised Visual Representation Learning (2019, background): “Similarly, when learning representation from videos, one can either make use of the synchronized cross-modality stream of audio, video, and potentially subtitles [35, 39, 23, 44], or of the consistency in the temporal dimension [41].”
Objects that Sound (2017, methodology): “Finally, a similar technique to ours appears in the concurrent work of [39], while later works of [40, 41] are also relevant.”
The Challenges of Continuous Self-Supervised Learning (2022, methodology): “coding [36, 54, 55], inpainting [59] and colorization [18, 34, 86], to methods that learn representations predictive of spatial context [19, 22, 53], temporal context [21, 47, 58, 61, 79, 80], or concurrent modalities like audio [4, 52, 57, 70], text [17, 23, 62] or speech [43, 44].”
Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization (2018, methodology): “Our method achieves higher accuracy than [32] both when both methods rely on the video-stream only (85.8% vs 77.6”
Audio-Visual Instance Discrimination with Cross-Modal Agreement (2020, methodology): “We also compare Cross-AVID and CMA to prior work, including video-based self-supervised learning methods [29, 49, 54, 87], and methods that leverage the natural correspondence between audio and video [2, 41, 62].”
The Sound of Pixels (2018, background): “[17] demonstrates an audio-visual system that separates onscreen sound vs.”
Self-supervised audio representation learning for mobile devices (2019, background): “Learning multimodal representations : Several papers have recently investigated learning audio representations exploiting the correlation with other modalities, e.g., text [8], images [32] and videos [31, 16, 1, 22, 10].”
Self-Supervised Learning by Cross-Modal Audio-Video Clustering (2019, background): “Temporal synchronization [29, 44] requires constructing positive/negative examples corresponding to in-sync and out-of-sync video-audio pairs.”
Audio-Visual Model Distillation Using Acoustic Images (2019, background): “More recent works [2, 3, 9, 31, 38] train both visual and audio networks aiming at learning multimodal representations useful for many applications, such as cross-modal retrieval, speech separation, sound source localization, action recognition, and on/off-screen audio source separation.”
Self-supervised Audio-visual Co-segmentation (2019, methodology): “More recently, [1, 16, 11] used audio-visual correspondence to separate sound sources.”
SoundSpaces: Audio-Visual Navigation in 3D Environments (2019, background): “This includes interesting directions for synthesizing sounds for video [71,16,110], spatializing sound [65,32], sound source separation [109,31,70,26,33], cross-modal feature learning [106,107,72,30], AV tracking [34,8,9,2], and learning material properties [71].”