The Sound of Pixels

By Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh H. McDermott, Antonio Torralba. 2018

599 citations recorded by the source corpus.

We introduce PixelPlayer, a system that, by leveraging large amounts of unlabeled videos, learns to locate image regions which produce sounds and separate the input sounds into a set of components that represents the sound from each pixel. Our approach capitalizes on the natural synchronization of the visual and audio modalities to learn models that jointly parse sounds and images, without requiring additional manual supervision. Experimental results on a newly collected MUSIC dataset show that our proposed Mix-and-Separate framework outperforms several baselines on source separation. Qualitative results suggest our model learns to ground sounds in vision, enabling applications such as independently adjusting the volume of sound sources.

Subfields: Cognitive Neuroscience, Multimodal Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Multimodal Learning

In Signal Processing

Citations in the corpus

Learning Video Representations from Textual Web Supervision (2020, background): “Prior works have leveraged ambient sound [52, 5, 4, 71, 34, 51, 55], dialogue [49], and narration [70, 2, 72, 73, 46, 45, 3], all of which of which serve as useful signals.”
Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “[238] Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba.”
Contrastive Bidirectional Transformer for Temporal Representation Learning (2019, background): “Since videos contain both visual and audio signals that are synchronized, the two modalities can supervised each other [3, 18, 17, 37].”
Objects that Sound (2017, methodology): “Finally, a similar technique to ours appears in the concurrent work of [39], while later works of [40, 41] are also relevant.”
Audio-Visual Scene Analysis with Self-Supervised Multisensory Features (2018, background): “Work by Zhao et al. [48] and Gao et al. [49] separate sound for multiple visible objects (e.g. musical instruments).”
Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization (2018, neutral): “Our work is concurrent with the work of Zhao et al. [45] and that of Owens and Efros [32].”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, methodology): “Other approaches use self-supervised learning for audio-video applications, such as audio-visual source separation [14, 15, 55] and spatial audio generation [56–58].”
Looking to listen at the cocktail party (2018, neutral): “The threshold was set empirically using synthetic mixtures of clean speech and non-speech interfering noise at different, known SNR”
Learning Words by Drawing Images (2019, background): “Recently proposed models that learn these correspondences can be used to perform tasks such as visually-guided audio source separation, or localizing source of a sound within an image or video [35, 44, 42, 17, 1, 18, 5].”
2.5D Visual Sound (2018, methodology): “State-of-the-art systems [13, 52, 31, 1, 10] aim to separate a mixed monaural audio recording into its component sound sources, and thus far they rely solely on the spatial cues evident in the visual stream.”
Self-supervised Audio-visual Co-segmentation (2019, methodology): “The model is an extension of recently proposed work that maps image pixels to sounds [1].”
AudioCaps: Generating Captions for Audios in The Wild (2019, background): “…explored (Nagrani et al., 2017; Chung et al., 2018; Senocak et al., 2018; Afouras et al., 2018; Gao et al., 2018; Arand-jelovic and Zisserman, 2018; Zhao et al., 2018), along with blind source separation aided by visual features (Ephrat et al., 2018) and learning of audio-visual multisensory…”
Learning to Detect and Retrieve Objects From Unlabeled Videos (2019, background): “Another line of studies target Audio-Visual alignment in unlabeled videos [2, 3, 8, 11, 12, 18, 20] capitalizing on the natural synchronization of the visual and audio modalities to learn models that discover audio-visual correspondence.”
Vggsound: A Large-Scale Audio-Visual Dataset (2020, background): “Another important task is to separate the sound of speciﬁc objects as they appear in a given frame or video clip [18, 19].”
SCSampler: Sampling Salient Clips From Video for Efficient Action Recognition (2019, background): “Recent work [2, 3, 4, 15, 26, 35, 60] has shown that the audio channel provides strong cues about the content of the video and this semantic correlation can be leveraged for various applications.”
Self-Supervised Generation of Spatial Audio for 360 Video (2018, background): “However, instead of only separating human speakers [42] or musical instruments [45], we aim to separate multiple unidentified types of sound sources.”
Audio-Visual Embodied Navigation (2019, background): “This includes interesting directions for synthesizing sounds for video [73, 18, 116], spatializing monaural sounds [33, 67], sound source separation [32, 72, 115, 29, 34], cross-modal feature learning [112, 113, 74], audio-visual target tracking [35, 9, 10, 3], and learning material properties [73].”
Deep Audio-visual Learning: A Survey (2020, methodology): “Therefore, Zhao et al. [28] introduced a system called PixelPlayer that used a two-stream network and presented a mix-and-separate framework to train the entire network.”