Look, Listen and Learn

By Relja Arandjelovi'c, Andrew Zisserman. 2017

1,046 citations recorded by the source corpus.

We consider the question: what can be learnt by looking at and listening to a large number of unlabelled videos? There is a valuable, but so far untapped, source of information contained in the video itself – the correspondence between the visual and the audio streams, and we introduce a novel “Audio-Visual Correspondence” learning task that makes use of this. Training visual and audio networks from scratch, without any additional supervision other than the raw unconstrained videos themselves, is shown to successfully solve this task, and, more interestingly, result in good visual and audio representations. These features set the new state-of-the-art on two sound classification benchmarks, and perform on par with the state-of-the-art selfsupervised approaches on ImageNet classification. We also demonstrate that the network is able to localize objects in both modalities, as well as perform fine-grained recognition tasks.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Signal Processing

Citations in the corpus

Self-Supervised Learning of Video-Induced Visual Invariances (2019, background): “Multi-modal learning methods that combine vision and audio [42, 9, 46, 3], and vision and text [57] achieve better performance than unimodal baselines.”
Learning Representations by Maximizing Mutual Information Across Views (2019, background): “Closely related work considers learning representations by predicting cross-modal correspondence [Arandjelović and Zisserman, 2017, 2018]. In concurrent work, Tian et al. [2019] propose applying global Deep InfoMax across”
Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, background): “Beyond video, information about camera motion [1, 32], scene geometry [67], or sound [3, 4] can all serve as natural sources of supervision.”
Learning Video Representations from Textual Web Supervision (2020, background): “Prior works have leveraged ambient sound [52, 5, 4, 71, 34, 51, 55], dialogue [49], and narration [70, 2, 72, 73, 46, 45, 3], all of which of which serve as useful signals.”
Self-Supervised MultiModal Versatile Networks (2020, methodology): “For Vision and Audio, contrastive based loss consistently outperforms logistic loss (used in [4, 35]) by 2% on vision downstream tasks, and is on par for audio.”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, background): “sigmoid cross entropy loss) that has shown to be effective in video-audio matching [7, 8].”
Self-Supervised Learning of Pretext-Invariant Representations (2019, background): “For video data, these pretext tasks learn representations by ordering video frames [1, 18, 34, 41, 51, 79, 83], tracking [62, 77], or using cross-modal signals like audio [2, 3, 19, 36, 60, 61].”
A critical analysis of self-supervision, or what we can learn from a single image (2019, methodology): “…al., 2018), future-frame synthesis (Srivastava et al., 2015), audio prediction from video (de Sa, 1994; Owens et al., 2016), audio-video alignment (Arandjelovi´c & Zisserman, 2017), ego-motion estimation (Ja-yaraman & Grauman, 2015), slow feature analysis with higher order temporal coherence…”
Scaling and Benchmarking Self-Supervised Visual Representation Learning (2019, background): “Multi-modal information such as depth from a sensor [17], sound in a video [3, 4, 23, 53], sensors on an autonomous vehicle [2, 30, 79] etc.”
Scattering Networks for Hybrid Representation Learning (2018, methodology): “ment, we consider the linear classification task on ILSVRC 2012 often used to evaluate unsupervised representations [3].”
Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learning (2017, neutral): “Likewise, Aytar et al (2016) introduced a method for transferring object labels from visual CNNs to audio CNNs.”
Objects that Sound (2017, methodology): “There has been a recent surge of interest in cross-modal learning from images and audio [1–4].”
Self-labelling via simultaneous clustering and representation learning (2019, methodology): “…al., 2018), future-frame synthesis (Srivastava et al., 2015), audio prediction from video (de Sa, 1994; Owens et al., 2016), audio-video alignment (Arandjelovi´c & Zisserman, 2017), ego-motion estimation (Jayaraman & Grauman, 2015), slow feature analysis with higher order temporal coherence…”
The Challenges of Continuous Self-Supervised Learning (2022, methodology): “The training data stream is created by sampling images at random from the four datasets {Dp1 , Dp2 , Dp3 , Dp4 } where [ p1 , p2 , p3 , p4] is a permutation of the sequence [1 , 2 , 3 , 4] .”
Transformers in Vision: A Survey (2021, methodology): “, text & image [121], audio & video [9, 105] or RGB & low [173].”
Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning (2018, methodology): “In a visual feature extraction task, x could be images and u related audio or text (Arandjelovic and Zisserman, 2017).”
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text (2021, methodology): “Inspired by [1, 4, 64], we use Noise-ContrastiveEstimation (NCE) to align video-text and video-audio pairs.”
Audio-Visual Scene Analysis with Self-Supervised Multisensory Features (2018, methodology): “Next, we trained our multisensory network with the self-supervision task of [16] rather than our own, i.”
Learning a Text-Video Embedding from Incomplete and Heterogeneous Data (2018, background): “Other more recent works [34], [39] explore appearance-audio two-stream architectures for video representation.”