Ambient Sound Provides Supervision for Visual Learning

By Andrew Owens, Jiajun Wu, Josh H. McDermott, William T. Freeman, Antonio Torralba. 2016

328 citations recorded by the source corpus.

The sound of crashing waves, the roar of fast-moving cars – sound conveys important information about the objects in our surroundings. In this work, we show that ambient sounds can be used as a supervisory signal for learning visual models. To demonstrate this, we train a convolutional neural network to predict a statistical summary of the sound associated with a video frame. We show that, through this process, the network learns a representation that conveys information about objects and scenes. We evaluate this representation on several recognition tasks, finding that its performance is comparable to that of other state-of-the-art unsupervised learning methods. Finally, we show through visualizations that the network learns units that are selective to objects that are often associated with characteristic sounds.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

Self-Supervised MultiModal Versatile Networks (2020, background): “Cross-modal teacher-student methods [10, 62] exploit the temporal co-occurrence between visual and audio modalities in a video to learn good representations.”
Self-Supervised Learning of Visual Features through Embedding Images into Text Topic Spaces (2017, neutral): “As an additional experiment we have calculated the clas-siﬁcation performance of the combination of TextTopicNet (Wiki) and Sound entries in Table 2.”
Self-Supervised Learning of Pretext-Invariant Representations (2019, background): “For video data, these pretext tasks learn representations by ordering video frames [1, 18, 34, 41, 51, 79, 83], tracking [62, 77], or using cross-modal signals like audio [2, 3, 19, 36, 60, 61].”
Multi-task Self-Supervised Visual Learning (2017, background): “In the last few years, a considerable number of such tasks have been proposed [1, 2, 6, 7, 8, 17, 20, 21, 23, 25, 26, 27, 28, 29, 31, 39, 40, 42, 43, 46, 47], such as asking a neural network to colorize grayscale images, ﬁll in image holes, solve jigsaw puzzles made from image patches, or predict…”
Scaling and Benchmarking Self-Supervised Visual Representation Learning (2019, methodology): “sound [53] and methods that use only the visual data (images, videos).”
Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization (2018, methodology): “Audio signals were also used to pre-train visual models [44].”
Learning to Perform Physics Experiments via Deep Reinforcement Learning (2016, neutral): “…for example synthesizing sounds from visual images (Owens et al., 2015), using summary statistics of audio to learn features for object recognition (Owens et al., 2016) or image colorization (Zhang et al., 2016). of (Goodfellow et al., 2014), and more recently autoregressive models have been very…”
Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input (2018, background): “Recent work has focused on cross modal learning between vision and sounds [3, 4, 36, 37].”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, background): “, sight and sound of musical instruments) as a signal for selfsupervised learning of both audio and visual features [37–42].”
Self-Supervised Feature Learning by Learning to Spot Artifacts (2018, background): “Other sources of natural supervision include ambient sound [30], temporal ordering [27] and physical interaction [33].”
Audio-Visual Instance Discrimination with Cross-Modal Agreement (2020, methodology): “Examples include the motion and appearance of a video [30, 80], depth and appearance [36, 91], luminance and chrominance of an image [80, 91], or as in our work sound and video [2, 4, 12, 63].”
The Sound of Pixels (2018, methodology): “For example, [29] used audio to supervise visual representations, [30,3] used vision to supervise audio representations, and [15] used sound and vision to jointly supervise each other.”
Geometry Guided Convolutional Neural Networks for Self-Supervised Video Representation Learning (2018, methodology): “Prior work on self-supervised learning uses the learned neural networks as the initialization for a fine-tuning stage for a particular task, such as object detection [7, 37], scene classification [26], and video recognition [24, 37].”