Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learning

By Andrew Owens, Jiajun Wu, Josh H. McDermott, William T. Freeman, Antonio Torralba. 2017

169 citations recorded by the source corpus.

The sound of crashing waves, the roar of fast-moving cars—sound conveys important information about the objects in our surroundings. In this work, we show that ambient sounds can be used as a supervisory signal for learning visual models. To demonstrate this, we train a convolutional neural network to predict a statistical summary of the sound associated with a video frame. We show that, through this process, the network learns a representation that conveys information about objects and scenes. We evaluate this representation on several recognition tasks, finding that its performance is comparable to that of other state-of-the-art unsupervised learning methods. Finally, we show through visualizations that the network learns units that are selective to objects that are often associated with characteristic sounds. This paper extends an earlier conference paper, Owens et al. (in: European conference on computer vision, 2016b), with additional experiments and discussion.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

Self-Supervised Learning of Video-Induced Visual Invariances (2019, background): “Multi-modal learning methods that combine vision and audio [42, 9, 46, 3], and vision and text [57] achieve better performance than unimodal baselines.”
VideoBERT: A Joint Model for Video and Language Representation Learning (2019, background): “Since most videos contain synchronized audio and visual signals, the two modalities can supervise each other to learn strong self-supervised video representations [3, 20, 21].”
Learning Video Representations from Textual Web Supervision (2020, background): “Prior works have leveraged ambient sound [52, 5, 4, 71, 34, 51, 55], dialogue [49], and narration [70, 2, 72, 73, 46, 45, 3], all of which of which serve as useful signals.”
Learning Features by Watching Objects Move (2016, background): “Other pretext tasks include predicting color channels from luminance [25, 51] or vice versa [52], and predicting sounds from video frames [7,33].”
Colorful Image Colorization (2016, methodology): “resentations in this way goes back at least to autoencoders [6]. More recent works have explored feature learning via data imputation, where a held-out subset of the complete data is predicted (e.g., [7,8,9,10,11,12,13]). Our method follows in this line, and can be termed a cross-channel encoder. We”
Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction (2016, methodology): “Figures 3(a) and (b) shows our split brain autoen-coder method compared to previous self-supervised meth-ods [4, 38, 28, 42, 5, 27] on the ImageNet and Places classi-ﬁcation tests, respectively.”
Generating Videos with Scene Dynamics (2016, background): “These visualizations suggest that scaling up future generation might be a promising supervisory signal for object recognition and complementary to [27, 5, 46].”
Deep Clustering for Unsupervised Learning of Visual Features (2018, background): “Appart from spatial and temporal coherence, many other signals have been explored: image colorization [33, 71], cross-channel prediction [72], sound [44] or instance counting [43].”
Contrastive Bidirectional Transformer for Temporal Representation Learning (2019, background): “Since videos contain both visual and audio signals that are synchronized, the two modalities can supervised each other [3, 18, 17, 37].”
Look, Listen and Learn (2017, background): “However, their goal is always to train a single network for one of the modes, for example, train a visual network to generate sounds in [20, 21]; or train an audio network to correlate with visual outputs in [2, 10], where the visual networks are pre-trained and fixed and act as a teacher.”
Colorization as a Proxy Task for Visual Understanding (2017, background): “[29], somewhat breaking from the temporal category, operate on a single video frame to predict a statistical summary of the audio from the entire clip.”
Objects that Sound (2017, background): “There has been a recent surge of interest in cross-modal learning from images and audio [3, 4, 16, 24].”
Network Dissection: Quantifying Interpretability of Deep Visual Representations (2017, methodology): “For self-supervised training tasks, we select several recent models trained on predicting context (context) [9], solving puzzles (puzzle) [21], predicting ego-motion (egomotion) [14], learning by moving (moving) [1], predicting video frame order (videoorder) [18] or tracking (tracking) [33], detecti”
Interpreting Deep Visual Representations via Network Dissection (2017, methodology): “For example, CNNs have been trained by predicting image context [23], by colorizing gray images [28], [29], by solving image puzzle [24], and by associating the images with ambient sounds [30].”
Unsupervised Representation Learning by Sorting Sequences (2017, methodology): “In addition to using only individual images, several recent directions have been explored by grouping visual entities using co-occurrence in space and time [13], using graph-based constraints [22], and crossmodal supervision from sounds [29].”
Audio-Visual Scene Analysis with Self-Supervised Multisensory Features (2018, methodology): “While at first glance this may seem low, we note that in many videos the sounds occur off-screen [15].”
Learning Problem-agnostic Speech Representations from Multiple Self-supervised Tasks (2019, background): “Some attempts have also been done to extend self-supervised learning to different modalities [12,13] or to audio representations only [14–17].”
Playing hard exploration games by watching YouTube (2018, methodology): “This idea is motivated by several recent works in the self-supervised feature learning literature [3, 11, 12, 28, 42, 45, 26].”
Self-supervised audio representation learning for mobile devices (2019, background): “Learning multimodal representations : Several papers have recently investigated learning audio representations exploiting the correlation with other modalities, e.g., text [8], images [32] and videos [31, 16, 1, 22, 10].”