Learning to Localize Sound Source in Visual Scenes

By Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, In-So Kweon. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018

389 citations recorded by the source corpus.

Visual events are usually accompanied by sounds in our daily lives. We pose the question: Can the machine learn the correspondence between visual scene and the sound, and localize the sound source only by observing sound and visual scene pairs like human? In this paper, we propose a novel unsupervised algorithm to address the problem of localizing the sound source in visual scenes. A two-stream network structure which handles each modality, with attention mechanism is developed for sound source localization. Moreover, although our network is formulated within the unsupervised learning framework, it can be extended to a unified architecture with a simple modification for the supervised and semi-supervised learning settings as well. Meanwhile, a new sound source dataset is developed for performance evaluation. Our empirical evaluation shows that the unsupervised method eventually go through false conclusion in some cases. We also show that even with a few supervision, i.e., semi-supervised setup, false conclusion is able to be corrected effectively.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/b91d738cd1f5d550c5b27f328e55308a0a73b2d2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Look, Listen, and Act: Towards Audio-Visual Embodied Navigation (2019, background): “More recently, researchers have proposed to train a deep neural network to see and hear many unlabeled videos to localize the objects that sound [26], [27], [28], [29], [30], [31], [32], [33],”
Self-Supervised Moving Vehicle Tracking With Stereo Sound (2019, background): “Recent approaches [4, 35, 40, 18] have trained a deep neural network to measure the correlations between visual and sound, and then used network localization techniques to locate the sound source in images.”
Audio-Visual Embodied Navigation (2019, background): “Unlike prior work that localizes pixels in video frames associated with sounds [92, 84, 7, 47], our goal is to learn navigation policies to actively locate an audio target in a 3D environment.”
Broaden Your Views for Self-Supervised Video Learning (2021, positive): “A simple but effective approach to train representations consists in classifying whether a video clip and an audio sample correspond to each other [6, 7, 45, 60, 70].”
Deep Audio-visual Learning: A Survey (2020, background): “[26] learned to localize sound sources by merely watching and listening to videos.”
Audio-Visual Event Localization in Unconstrained Videos (2018, neutral): “Concurrently, some interesting and related works on sound source separation, localization and audio-visual representation learning are explored in [13,16,40,47,59].”
Audio-Visual Instance Discrimination with Cross-Modal Agreement (2020, background): “In particular, audio self-supervision has shown to be useful for localization [3, 71], lip-speech synchronization [12] and visual representation learning [2, 36, 58] and audio spatialization [52].”
The Sound of Pixels (2018, background): “In work related to our paper, [31,32] studied how to localize sounds in vision, however they do not separate multiple sounds from a mixed signal.”
AudioCaps: Generating Captions for Audios in The Wild (2019, background): “The task of localizing the sound source within the visual input have been actively explored (Nagrani et al., 2017; Chung et al., 2018; Senocak et al., 2018; Afouras et al., 2018; Gao et al., 2018; Arand-jelovic and Zisserman, 2018; Zhao et al., 2018), along with blind source separation aided by…”
Learning Words by Drawing Images (2019, background): “Recently proposed models that learn these correspondences can be used to perform tasks such as visually-guided audio source separation, or localizing source of a sound within an image or video [35, 44, 42, 17, 1, 18, 5].”
Multi-modal Self-Supervision from Generalized Data Transformations (2020, methodology): “Futhermore, multi-modal learning has been used for several applications such as lip reading [4, 20, 21], audio-visual source separation and localization [3, 7, 28, 29, 39, 77, 87, 91, 113, 114], speech recognition [2, 73, 84], efficient inference [30, 56], egocentric action recognition [51] and navi”
Audio-Visual Scene Analysis with Self-Supervised Multisensory Features (2018, background): “This is in contrast to single-frame methods models [50,52,16], which largely attend to sound-making objects rather than actions.”
Empowering Things With Intelligence: A Survey of the Progress, Challenges, and Opportunities in Artificial Intelligence of Things (2020, background): “stream network with an attention mechanism is proposed to directly localize a sound source in images [249].”
SoundSpaces: Audio-Visual Navigation in 3D Environments (2019, background): “Unlike prior work that localizes pixels in video frames associated with sounds [88,82,6,45], our goal is to learn navigation policies for agents to actively”
VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency (2021, background): “Apart from source separation, recent inspiring work integrates both audio and visual cues on an array of other tasks including self-supervised representation learning [57, 6, 8, 55, 41, 23], localizing sounds in video frames [7, 66, 71, 36], generating sounds from video [56, 84, 25, 47, 83], and act”
Co-Separating Sounds of Visual Objects (2019, background): “Localizing Sounds in Video Frames Localization entails identifying the pixels where the sound of a video comes from, but not separating the audio [25, 19, 10, 4, 40, 45].”
Audiovisual SlowFast Networks for Video Recognition (2020, background): “Other audiovisual tasks that have been studied include audio-visual speech recognition [54, 50], sound-source localization [3, 59], audio-visual source separation [52, 19], and audiovisual question answering [1].”
Coincidence, Categorization, and Consolidation: Learning to Recognize Sounds with Minimal Supervision (2019, background): “…(DeepCCA) [26], a deep learning generalization of linear CCA; (ii) Look, Listen, and Learn ( L 3 ) [12], which learns representations that can predict audio-visual frame correspondence in videos; and (iii) metric learning losses applied to shared multi-modal embedding spaces [27, 28, 29, 30].”