Self-supervised Audio-visual Co-segmentation

By Andrew Rouditchenko, Hang Zhao, Chuang Gan, Josh H. McDermott, Antonio Torralba. 2019

106 citations recorded by the source corpus.

Segmenting objects in images and separating sound sources in audio are challenging tasks, in part because traditional approaches require large amounts of labeled data. In this paper we develop a neural network model for visual object segmentation and sound source separation that learns from natural videos through self-supervision. The model is an extension of recently proposed work that maps image pixels to sounds [1]. Here, we introduce a learning approach to disentangle concepts in the neural networks, and assign semantic categories to network feature channels to enable independent image segmentation and sound source separation after audio-visual training on videos. Our evaluations show that the disentangled model outperforms several baselines in semantic segmentation and sound source separation.

Subfields: Cognitive Neuroscience, Multimodal Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Multimodal Learning

In Signal Processing

Citations in the corpus

Learning Video Representations from Textual Web Supervision (2020, background): “Prior works have leveraged ambient sound [52, 5, 4, 71, 34, 51, 55], dialogue [49], and narration [70, 2, 72, 73, 46, 45, 3], all of which of which serve as useful signals.”
Look, Listen, and Act: Towards Audio-Visual Embodied Navigation (2019, background): “network to see and hear many unlabeled videos to localize the objects that sound [24], [25], [?], [26], [27], [28], [29], [30], [31], [32].”
Multi-modal Self-Supervision from Generalized Data Transformations (2020, methodology): “Futhermore, multi-modal learning has been used for several applications such as lip reading [4, 20, 21], audio-visual source separation and localization [3, 7, 28, 29, 39, 77, 87, 91, 113, 114], speech recognition [2, 73, 84], efficient inference [30, 56], egocentric action recognition [51] and navi”
Self-Supervised Learning of Audio-Visual Objects from Video (2020, methodology): “[26, 28] use semantic object detectors trained on instrument categories, while [51, 58] do not explicitly group a scene”
Self-Supervised Moving Vehicle Tracking With Stereo Sound (2019, methodology): “[40, 39, 13, 17, 32] used the visual-audio correspondence to separate sound sources.”
VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency (2021, background): “Recent deep learning approaches focus on either musical instruments [24, 78, 74, 77, 62, 21] or speech [20, 2, 18, 52, 3, 15].”
An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation (2020, background): “[218] extended the work in [288] for unsynchronised audio and video data.”
Deep Audio-visual Learning: A Survey (2020, methodology): “[30] tried to perform localization and separation tasks using only video frames or sound by disentangling concepts learned by neural networks.”
Self-Supervised Learning by Cross-Modal Audio-Video Clustering (2019, background): “Other approaches learn from unlabeled multi-modal data for a speciﬁc target task, e.g . sound source localization [75] and audio-visual co-segmentation [54].”