Deep Audio-visual Learning: A Survey

By Hao Zhu, Mandi Luo, Rui Wang, Aihua Zheng, Ran He. 2020

194 citations recorded by the source corpus.

Audio-visual learning, aimed at exploiting the relationship between audio and visual modalities, has drawn considerable attention since deep learning started to be used successfully. Researchers tend to leverage these two modalities to improve the performance of previously considered single-modality tasks or address new challenging problems. In this paper, we provide a comprehensive survey of recent audio-visual learning development. We divide the current audio-visual learning tasks into four different subfields: audio-visual separation and localization, audio-visual correspondence learning, audio-visual generation, and audio-visual representation learning. State-of-the-art methods, as well as the remaining challenges of each subfield, are further discussed. Finally, we summarize the commonly used datasets and challenges.

Subfields: Cognitive Neuroscience, Multimodal Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Multimodal Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “Relation to existing tasks The past few years have seen audio studied in computer vision tasks [245] for action classification [110,226], object categorization [125,234], source localization and tracking [14, 197, 212] and embodied navigation [33].”
Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation (2021, background): “Driving talking faces with audio input [7, 78] has long been important research interest in both computer vision and graphics, where structural information and stitching techniques play a crucial role [4, 3, 57, 56, 76].”
An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation (2020, background): “[289] provided a bird’s-eye view of several AV tasks, to which deep learning has been applied.”
Sound-Guided Semantic Image Manipulation (2021, background): “Several studies [7, 18, 30, 36, 47, 52] have attempted to visualize the meaning of sound, but it is still challenging to reflect sound events in high-resolution images due to two reasons.”
VLP: A Survey on Vision-language Pre-training (2022, background): “Most previous works on multi-modal pre-training emphasize the joint modeling of language and vision but ignore the information buried in audios [138, 139].”
CelebV-HQ: A Large-Scale Video Facial Attributes Dataset (2022, background): “As stated before, VoxCeleb2 [14] is one of the largest in-the-wild face video datasets, and it contains massive face videos, that have contributed to the development of many fields [19,103,23,98,10,102,99,38].”