Co-Separating Sounds of Visual Objects

By Ruohan Gao, Kristen Grauman. 2019

224 citations recorded by the source corpus.

Learning how objects sound from video is challenging, since they often heavily overlap in a single audio channel. Current methods for visually-guided audio source separation sidestep the issue by training with artificially mixed video clips, but this puts unwieldy restrictions on training data collection and may even prevent learning the properties of "true" mixed sounds. We introduce a co-separation training paradigm that permits learning object-level sounds from unlabeled multi-source videos. Our novel training objective requires that the deep neural network's separated audio for similar-looking objects be consistently identifiable, while simultaneously reproducing accurate video-level audio tracks for each source training pair. Our approach disentangles sounds in realistic test videos, even in cases where an object was not observed individually during training. We obtain state-of-the-art results on visually-guided audio source separation and audio denoising for the MUSIC, AudioSet, and AV-Bench datasets.

Subfields: Cognitive Neuroscience, Multimodal Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Multimodal Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, background): “This idea has been further developed for visually-guided audio source separation [43–48], sound generation from silent videos [39, 49], sound localization in video frames [50, 51], face and voice association [52, 53], and video-based audio localization [54, 55].”
Multi-modal Self-Supervision from Generalized Data Transformations (2020, methodology): “Futhermore, multi-modal learning has been used for several applications such as lip reading [4, 20, 21], audio-visual source separation and localization [3, 7, 28, 29, 39, 77, 87, 91, 113, 114], speech recognition [2, 73, 84], efficient inference [30, 56], egocentric action recognition [51] and navi”
VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency (2021, methodology): “To generate training examples, we follow the commonly adopted “mix-and-separate” paradigm [19, 55, 2, 81, 26] and generate synthetic audio mixtures by mixing human speech segments.”
Audio-Visual Embodied Navigation (2019, background): “This includes interesting directions for synthesizing sounds for video [73, 18, 116], spatializing monaural sounds [33, 67], sound source separation [32, 72, 115, 29, 34], cross-modal feature learning [112, 113, 74], audio-visual target tracking [35, 9, 10, 3], and learning material properties [73].”
SoundSpaces: Audio-Visual Navigation in 3D Environments (2019, background): “This includes interesting directions for synthesizing sounds for video [71,16,110], spatializing sound [65,32], sound source separation [109,31,70,26,33], cross-modal feature learning [106,107,72,30], AV tracking [34,8,9,2], and learning material properties [71].”
An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation (2020, background): “Hence, Gao and Grauman [70] introduced a new training paradigm, called co-separation, in which an association between consistent sounds and visual objects across pairs of training videos was learned.”
Self-Supervised Learning of Audio-Visual Objects from Video (2020, methodology): “[25, 27] use semantic object detectors trained on instrument categories, while [53, 60] do not explicitly group a scene into objects and instead either pool the visual features or produce a per-pixel map that associates each pixel with a separated audio source.”
Listen to Look: Action Recognition by Previewing Audio (2019, background): “Recent work uses audio for an array of video understanding tasks outside of action recognition, including self-supervised representation learning [43, 5, 7, 41, 33, 54], audio-visual source separation [41, 2, 13, 19, 76, 21], localizing sounds in video frames [6, 48, 55], and generating sounds from ”