Visually Indicated Sounds

By Andrew Owens, Phillip Isola, Josh H. McDermott, Antonio Torralba, Edward H. Adelson, William T. Freeman. 2015

424 citations recorded by the source corpus.

Objects make distinctive sounds when they are hit or scratched. These sounds reveal aspects of an object's material properties, as well as the actions that produced them. In this paper, we propose the task of predicting what sound an object makes when struck as a way of studying physical interactions within a visual scene. We present an algorithm that synthesizes sound from silent videos of people hitting and scratching objects with a drumstick. This algorithm uses a recurrent neural network to predict sound features from videos and then produces a waveform from these features with an example-based synthesis procedure. We show that the sounds predicted by our model are realistic enough to fool participants in a "real or fake" psychophysical experiment, and that they convey significant information about material properties and physical interactions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Signal Processing

Citations in the corpus

What makes ImageNet good for transfer learning? (2016, background): “Attempted self-supervised tasks include predictions of ego-motion [1, 20], spatial context [11, 33, 30], temporal context [45], and even color [51] and sound [32].”
VideoBERT: A Joint Model for Video and Language Representation Learning (2019, background): “Since most videos contain synchronized audio and visual signals, the two modalities can supervise each other to learn strong self-supervised video representations [3, 20, 21].”
Contrastive Multiview Coding (2019, methodology): “The predictive approach has been extensively used in representation learning, for example, colorization [84, 85] and predicting sound from vision [58].”
Colorful Image Colorization (2016, background): “Using an off-the-shelf classifier to assess the realism of synthesized data has been previously suggested by [12].”
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric (2018, methodology): “In Table 5, we explore how well our perceptual task correlates to semantic tasks on the PASCAL dataset [15], using results summarized in [63], including additional selfsupervised methods [1, 43, 12, 56, 62, 41].”
Image-to-Image Translation with Conditional Adversarial Networks (2016, methodology): “This metric is similar to the “inception score” from [49], the object detection evaluation in [52], and the “semantic interpretability” measures in [58] and [39].”
A critical analysis of self-supervision, or what we can learn from a single image (2019, background): “, 2015), audio prediction from video (de Sa, 1994; Owens et al., 2016), audio-video alignment (Arandjelović & Zisserman, 2017), ego-motion estimation (Jayaraman & Grauman, 2015), slow feature analysis with higher order temporal coherence (Jayaraman & Grauman, 2016), transformation between frames (Ag”
Contrastive Bidirectional Transformer for Temporal Representation Learning (2019, background): “Since videos contain both visual and audio signals that are synchronized, the two modalities can supervised each other [3, 18, 17, 37].”
Look, Listen and Learn (2017, background): “However, their goal is always to train a single network for one of the modes, for example, train a visual network to generate sounds in [20, 21]; or train an audio network to correlate with visual outputs in [2, 10], where the visual networks are pre-trained and fixed and act as a teacher.”
Time-Contrastive Networks: Self-Supervised Learning from Video (2017, background): “For example, [23, 24] used co-occurrence of sounds and visual cues in videos to learn meaningful visual features.”
Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learning (2017, positive): “Our approach is closely related to recent audio-visual work (Owens et al 2016a) that predicts soundtracks for videos that show a person striking objects with a drumstick.”
Objects that Sound (2017, background): “In this paper our interest is in cross-modal learning from images and audio [4, 5, 16, 18, 23, 24].”
Generative Image Modeling Using Style and Structure Adversarial Networks (2016, background): “Some examples of these auxiliary tasks include predicting: the relative location of two patches [2], ego-motion in videos [15,16], physical signals [17,18,19].”
SoundNet: Learning Sound Representations from Unlabeled Video (2016, background): “Cross-Modal Learning and Unlabeled Video: Our approach is broadly inspired by efforts to model cross-modal relations [24, 14, 7, 26] and works that leverage large amounts of unlabeled video [25, 41, 8, 40, 39].”
Self-labelling via simultaneous clustering and representation learning (2019, background): “, 2015), audio prediction from video (de Sa, 1994; Owens et al., 2016), audio-video alignment (Arandjelović & Zisserman, 2017), ego-motion estimation (Jayaraman & Grauman, 2015), slow feature analysis with higher order temporal coherence (Jayaraman & Grauman, 2016), transformation between frames (Ag”
Audio-Visual Scene Analysis with Self-Supervised Multisensory Features (2018, background): “Starting with the early work of de Sa [3], there have been many self-supervised methods that learn to find correlations between sight and sound [13,14,15,16].”
Ambient Sound Provides Supervision for Visual Learning (2016, background): “This style of learning has been called “self supervision” [4] or “natural supervision” [30].”
Learning to Perform Physics Experiments via Deep Reinforcement Learning (2016, neutral): “Researchers have looked at cross modal learning, for example synthesizing sounds from visual images (Owens et al., 2015), using summary statistics of audio to learn features for object recognition (Owens et al., 2016) or image colorization (Zhang et al., 2016). of (Goodfellow et al., 2014), and…”
Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input (2018, background): “Recent work has focused on crossmodal learning between vision and sounds (Arandjelovic and Zisserman 2017; Aytar et al. 2016; Owens et al. 2016a, b).”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, background): “al for selfsupervised learning of both audio and visual features [37–42]. This idea has been further developed for visually-guided audio source separation [43–48], sound generation from silent videos [39, 49], sound localization in video frames [50, 51], face and voice association [52, 53], and vide”