2.5D Visual Sound

By Ruohan Gao, Kristen Grauman. 2018

151 citations recorded by the source corpus.

Binaural audio provides a listener with 3D sound sensation, allowing a rich perceptual experience of the scene. However, binaural recordings are scarcely available and require nontrivial expertise and equipment to obtain. We propose to convert common monaural audio into binaural audio by leveraging video. The key idea is that visual frames reveal significant spatial cues that, while explicitly lacking in the accompanying single-channel audio, are strongly linked to it. Our multi-modal approach recovers this link from unlabeled video. We devise a deep convolutional neural network that learns to decode the monaural (single-channel) soundtrack into its binaural counterpart by injecting visual information about object and scene configurations. We call the resulting output 2.5D visual sound---the visual stream helps "lift" the flat single channel audio into spatialized sound. In addition to sound generation, we show the self-supervised representation learned by our network benefits audio-visual source separation. Our video results: http://vision.cs.utexas.edu/projects/2.5D_visual_sound/

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “Meanwhile, visual information is increasingly used in historically audio-only tasks like speech transcription, voice recognition, audio spatialization [5,80,104,161], speaker diarization [10,83], and source separation [57,78,82].”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, background): “ded audio source separation [43–48], sound generation from silent videos [39, 49], sound localization in video frames [50, 51], face and voice association [52, 53], and video-based audio localization [54, 55]. While our work has a similar focus on audio and video, instead of learning the source of a”
Artificial intelligence in the creative industries: a review (2020, background): “With this knowledge, Gao and Grauman (2019) created binaural audio from a mono signal driven by the subject’s visual environment to enrich the perceptual experience of the scene.”
An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation (2020, background): “Hence, Gao and Grauman [70] introduced a new training paradigm, called co-separation, in which an association between consistent sounds and visual objects across pairs of training videos was learned.”
Audio-Visual Embodied Navigation (2019, background): “This includes interesting directions for synthesizing sounds for video [73, 18, 116], spatializing monaural sounds [33, 67], sound source separation [32, 72, 115, 29, 34], cross-modal feature learning [112, 113, 74], audio-visual target tracking [35, 9, 10, 3], and learning material properties [73].”
VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency (2021, background): “Apart from source separation, recent inspiring work integrates both audio and visual cues on an array of other tasks including self-supervised representation learning [57, 6, 8, 55, 41, 23], localizing sounds in video frames [7, 66, 71, 36], generating sounds from video [56, 84, 25, 47, 83], and act”
Self-Supervised Learning of Audio-Visual Objects from Video (2020, background): “In a related line of work, [24] distill visual object detectors into an audio model using stereo sound, while [26] use spatial information in a scene to convert mono sound to stereo.”
Learning to Localize Sound Sources in Visual Scenes: Analysis and Applications (2019, methodology): “Although the sound localization task is not effectively addressedwith our unsupervised learning approach with static images and mono audios, other methods that use spatial microphones [25], [53], [54], [55] or”
Co-Separating Sounds of Visual Objects (2019, methodology): “We find spectrogram masks to work better than direct prediction of spectrograms or raw waveforms for source separation, confirming reports in [49, 8, 13].”
Audio-Visual Model Distillation Using Acoustic Images (2019, background): “FAIR-Play [11] is an unlabelled video dataset with binaural audio that mimics human hearing.”
EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition (2019, background): “These fall in one of four categories: i) audio-visual representation learning [2, 5, 6, 23, 28, 29], ii) sound-source localisation [3, 28, 34], iii) audio-visual source separation [11, 28] and (iv) visual-question answering [1].”
Listen to Look: Action Recognition by Previewing Audio (2019, background): “Recent work uses audio for an array of video understanding tasks outside of action recognition, including self-supervised representation learning [43, 5, 7, 41, 33, 54], audio-visual source separation [41, 2, 13, 19, 76, 21], localizing sounds in video frames [6, 48, 55], and generating sounds from ”
Parameter Efficient Multimodal Transformers for Video Representation Learning (2020, background): “Recent approaches leverage unlabeled videos from speciﬁc domains (Owens et al., 2016; Gao & Grauman, 2019; Zhao et al., 2018; Ephrat et al., 2018; Alwassel et al., 2019; Miech et al., 2020; Piergiovanni et al., 2020) and often demonstrate on audio-visual source separation, localization, and…”