Learning to Separate Object Sounds by Watching Unlabeled Video

By Ruohan Gao, Rogério Schmidt Feris, Kristen Grauman. 2018

298 citations recorded by the source corpus.

Perceiving a scene most fully requires all the senses. Yet modeling how objects look and sound is challenging: most natural scenes and events contain multiple objects, and the audio track mixes all the sound sources together. We propose to learn audio-visual object models from unlabeled video, then exploit the visual context to perform audio source separation in novel videos. Our approach relies on a deep multi-instance multi-label learning framework to disentangle the audio frequency bases that map to individual visual objects, even without observing/hearing those objects in isolation. We show how the recovered disentangled bases can be used to guide audio source separation to obtain better-separated, object-level sounds. Our work is the first to learn audio source separation from large-scale "in the wild" videos containing multiple audio sources per video. We obtain state-of-the-art results on visually-aided audio source separation and audio denoising. Our video results: this http URL

Subfields: Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Self-Supervised Learning of Pretext-Invariant Representations (2019, background): “For video data, these pretext tasks learn representations by ordering video frames [1, 18, 34, 41, 51, 79, 83], tracking [62, 77], or using cross-modal signals like audio [2, 3, 19, 36, 60, 61].”
Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “Meanwhile, visual information is increasingly used in historically audio-only tasks like speech transcription, voice recognition, audio spatialization [5,80,104,161], speaker diarization [10,83], and source separation [57,78,82].”
Audio-Visual Scene Analysis with Self-Supervised Multisensory Features (2018, background): “[49] separate sound for multiple visible objects (e.”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, background): “This idea has been further developed for visually-guided audio source separation [43–48], sound generation from silent videos [39, 49], sound localization in video frames [50, 51], face and voice association [52, 53], and video-based audio localization [54, 55].”
AudioCaps: Generating Captions for Audios in The Wild (2019, background): “…source within the visual input have been actively explored (Nagrani et al., 2017; Chung et al., 2018; Senocak et al., 2018; Afouras et al., 2018; Gao et al., 2018; Arand-jelovic and Zisserman, 2018; Zhao et al., 2018), along with blind source separation aided by visual features (Ephrat et al.,…”
Uncertainty-Aware Audiovisual Activity Recognition Using Deep Bayesian Variational Inference (2018, background): “Recent work on sound localization and separation [14, 34] has shown the beneﬁts of a joint audiovisual representation for cross-modal self-supervised learning using only audio-visual correspondence as the objective function.”
ClusterFit: Improving Generalization of Visual Representations (2019, background): “These pre-text tasks can either be domain agnostic [5, 6, 30, 45, 60, 61] or exploit domain-specific information like spatial structure in images [13, 21, 42–44], color [12, 34, 35, 63, 64], illumination [14], temporal structure [16, 26, 37, 39, 40] or a co-occurring modality like sound [2, 3, 9, 19”
Grounding Spoken Words in Unlabeled Video (2019, background): “This idea has been further developed for visually-guided audio source separation [8,18,19], generating sounds from silent videos [17,21], localizing sounds in video frames [3, 19], learning association of faces and voices [13,15], and video-based audio spatialization [7,14].”
Self-supervised audio representation learning for mobile devices (2019, background): “Learning multimodal representations : Several papers have recently investigated learning audio representations exploiting the correlation with other modalities, e.g., text [8], images [32] and videos [31, 16, 1, 22, 10].”
An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation (2020, methodology): “In contrast to [68], the system could also perform visual localisation, which is the task of detecting the sound sources in the visual input.”
Steering Self-Supervised Feature Learning Beyond Local Pixel Statistics (2020, methodology): “Self-supervised learning has also been applied to other data domains such as video [50, 43, 48, 36] and audio [42, 57, 15].”
Audio-Visual Embodied Navigation (2019, background): “This includes interesting directions for synthesizing sounds for video [73, 18, 116], spatializing monaural sounds [33, 67], sound source separation [32, 72, 115, 29, 34], cross-modal feature learning [112, 113, 74], audio-visual target tracking [35, 9, 10, 3], and learning material properties [73].”
2.5D Visual Sound (2018, background): “State-of-the-art systems [13, 52, 31, 1, 10] aim to separate a mixed monaural audio recording into its component sound sources, and thus far they rely solely on the spatial cues evident in the visual stream.”
Audio-Visual Event Localization in Unconstrained Videos (2018, unknown): “Concurrently, some interesting and related works on sound source separation, localization and audio-visual representation learning are explored in [13,16,40,47,59].”
Listen to Look: Action Recognition by Previewing Audio (2019, background): “Recent work uses audio for an array of video understanding tasks outside of action recognition, including self-supervised representation learning [43, 5, 7, 41, 33, 54], audio-visual source separation [41, 2, 13, 19, 76, 21], localizing sounds in video frames [6, 48, 55], and generating sounds from ”