Deep Cross-Modal Audio-Visual Generation

By Lele Chen, Sudhanshu Srivastava, Zhiyao Duan, Chenliang Xu. 2017

235 citations recorded by the source corpus.

Cross-modal audio-visual perception has been a long-lasting topic in psychology and neurology, and various studies have discovered strong correlations in human perception of auditory and visual stimuli. Despite work on computational multimodal modeling, the problem of cross-modal audio-visual generation has not been systematically studied in the literature. In this paper, we make the first attempt to solve this cross-modal generation problem leveraging the power of deep generative adversarial training. Specifically, we use conditional generative adversarial networks to achieve cross-modal audio-visual generation of musical performances. We explore different encoding methods for audio and visual signals, and work on two scenarios: instrument-oriented generation and pose-oriented generation. Being the first to explore this new problem, we compose two new datasets with pairs of images and sounds of musical performances of different instruments. Our experiments using both classification and human evaluation demonstrate that our model has the ability to generate one modality, i.e., audio/visual, from the other modality, i.e., visual/audio, to a good extent. Our experiments on various design choices along with the datasets will facilitate future research in this new problem space.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

A Wavenet for Speech Denoising (2017, background): “Over the last several decades, machine learning has produced solutions to complex problems that were previously unattainable with signal processing techniques [4, 12, 38].”
SoundSpaces: Audio-Visual Navigation in 3D Environments (2019, background): “This includes interesting directions for synthesizing sounds for video [71,16,110], spatializing sound [65,32], sound source separation [109,31,70,26,33], cross-modal feature learning [106,107,72,30], AV tracking [34,8,9,2], and learning material properties [71].”
Lip Movements Generation at a Glance (2018, background): “Cross-modality generation has become an important and emerging topic of computer vision and its broader AI communities, where examples are beyond the most prominent image/video-to-text [10,19] and can be found in video-tosound [23], text-to-image [25], and even sound-to-image [4].”
Realistic Speech-Driven Facial Animation with GANs (2019, background): “[9] propose a GAN-based encoder-decoder architecture that uses CNNs in order to convert audio spectrograms to frames and vice versa.”
Deep Audio-visual Learning: A Survey (2020, methodology): “further proposed the Sub-URMP [72] dataset that contains multiple video frames and audio files extracted from the URMP dataset.”
Audio-Visual Embodied Navigation (2019, background): “This includes interesting directions for synthesizing sounds for video [73, 18, 116], spatializing monaural sounds [33, 67], sound source separation [32, 72, 115, 29, 34], cross-modal feature learning [112, 113, 74], audio-visual target tracking [35, 9, 10, 3], and learning material properties [73].”
Co-Separating Sounds of Visual Objects (2019, methodology): “Generating Sounds from Video Sound generation methods synthesize a sound track from a visual input [32, 54, 6].”
Sound-Guided Semantic Image Manipulation (2021, background): “Several studies [7, 18, 30, 36, 47, 52] have attempted to visualize the meaning of sound, but it is still challenging to reflect sound events in high-resolution images due to two reasons.”
Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation (2021, background): “Recently, with the development of end-to-end cross audio-visual generation [11, 70, 24, 69, 64, 74, 23, 54, 65, 22], researchers have explored the speaker-independent setting [14, 50, 45, 72, 75], which seeks a universal model that handles all identities with one or few frame references.”
Taming Visually Guided Sound Generation (2021, methodology): “images [13, 17, 35, 41, 44, 46, 50, 52, 64, 66, 67], videos [6, 12, 25, 34, 38, 42, 59, 60, 60, 63], and audios [1, 9, 15, 22, 24, 43, 57, 58], or separate sounds [18, 19, 69, 70, 74].”
Synthesizing Audio with Generative Adversarial Networks (2018, methodology): “Chen et al. (2017) use GANs to map images into associated audio spectrograms.”
Seeing through Sounds: Predicting Visual Semantic Segmentation Results from Multichannel Audio Signals (2019, background): “Some very recent studies have focused on cross-modal analysis between audio and visual information, such as visual feature learning from sounds [3, 4], sound prediction from silent videos [5], and cross-modal content generation [6].”
Audio-Visual Event Localization in Unconstrained Videos (2018, unknown): “Aside from works in representation learning, audio-visual cross-modal synthesis is studied in [7,39,40], and associations between natural image scenes and accompanying free-form spoken audio captions are explored in [37].”
Hierarchical Cross-Modal Talking Face Generation With Dynamic Pixel-Wise Loss (2019, background): “Modeling the dynamics of a moving human face/body conditioned on another modality is a fundamental problem in computer vision, where applications are ranging from audio-to-video generation [28, 3, 2] to text-to-video generation [23, 19] and to skeleton-to-image/video generation [21, 7].”
Learning to Separate Object Sounds by Watching Unlabeled Video (2018, background): “Recent work exploits recurrent networks [77] or conditional generative adversarial networks (C-GANs) [12] to generate audio from input video frames.”
2.5D Visual Sound (2018, background): “Recurrent networks [53] or conditional generative adversarial networks [7] can generate audio for input video frames, while powerful simulators can synthesize audio-visual data for 3D shapes [51].”
MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation (2022, methodology): “Only on U-Net scale of [2,3,4], video attention and cross-modal attention are applied, and the window size for cross-modal attention”