Audio-Visual Event Localization in Unconstrained Videos

By Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, Chenliang Xu. 2018

616 citations recorded by the source corpus.

In this paper, we introduce a novel problem of audio-visual event localization in unconstrained videos. We define an audio-visual event as an event that is both visible and audible in a video segment. We collect an Audio-Visual Event(AVE) dataset to systemically investigate three temporal localization tasks: supervised and weakly-supervised audio-visual event localization, and cross-modality localization. We develop an audio-guided visual attention mechanism to explore audio-visual correlations, propose a dual multimodal residual network (DMRN) to fuse information over the two modalities, and introduce an audio-visual distance learning network to handle the cross-modality localization. Our experiments support the following findings: joint modeling of auditory and visual modalities outperforms independent modeling, the learned attention can capture semantics of sounding objects, temporal alignment is important for audio-visual fusion, the proposed DMRN is effective in fusing audio-visual features, and strong correlations between the two modalities enable cross-modality localization.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Signal Processing

Citations in the corpus

Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “The past few years have seen audio studied in computer vision tasks [245] for action classification [110,226], object categorization [125,234], source localization and tracking [14,197,212] and embodied navigation [33].”
HMS: Hierarchical Modality Selection for Efficient Video Recognition (2021, methodology): “These multimodal inputs are used in other video tasks such as self-supervised learning in videos [1, 25], sound localization [32, 36] and generation [54, 55] from videos as well.”
Audio-Visual Embodied Navigation (2019, background): “Unlike prior work that localizes pixels in video frames associated with sounds [92, 84, 7, 47], our goal is to learn navigation policies to actively locate an audio target in a 3D environment.”
Self-supervised Audio-visual Co-segmentation (2019, methodology): “Furthermore, we substantially extend the scale of previous work [1] by training on a video dataset of naturally occurring audio-visual events with over 4000 videos [19].”
SoundSpaces: Audio-Visual Navigation in 3D Environments (2019, background): “Unlike prior work that localizes pixels in video frames associated with sounds [88,82,6,45], our goal is to learn navigation policies for agents to actively”
2.5D Visual Sound (2018, background): “Furthermore, different from the task of localizing pixels responsible for a given sound [21, 18, 54, 3, 52, 40, 44], our goal is to perform binaural audio synthesis.”
Balanced Multimodal Learning via On-the-fly Gradient Modulation (2022, positive): “We verify the effectiveness of our method on commonly used AVE [36] dataset.”
Cross-Modal learning for Audio-Visual Video Parsing (2021, background): “, [15]) that are formulated as an MIL problem where an audio-visual snippet pair is regarded as an instance, each audio snippet and the corresponding visual snippet occurred at the same time denote two individual instances in a bag in our task.”
Vggsound: A Large-Scale Audio-Visual Dataset (2020, background): “In addition, the authors of [21] manually curated a high-quality, but small dataset that guarantees audio-visual correspondence for multi-modal learning, where the objects or events that are the cause of a sound must also be observable in the visual stream.”
Audio-Visual Model Distillation Using Acoustic Images (2019, background): “Several recent works address audio-related tasks such as natural sound recognition [26], speech separation and enhancement [1, 10], audio event classification or sound source localization [35, 42], either by directly modeling raw audio signals with 1D convolutions [5, 31, 37] or, most popularly, by ”
Deep Audio-visual Learning: A Survey (2020, background): “Attention mechanisms cause the model to focus on the primary area[36].”
VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency (2021, background): “Apart from source separation, recent inspiring work integrates both audio and visual cues on an array of other tasks including self-supervised representation learning [57, 6, 8, 55, 41, 23], localizing sounds in video frames [7, 66, 71, 36], generating sounds from video [56, 84, 25, 47, 83], and act”
Learning to Localize Sound Sources in Visual Scenes: Analysis and Applications (2019, background): “[9] also focus on audio-visual event localization but with fully and weakly-supervised approaches.”
Listen to Look: Action Recognition by Previewing Audio (2019, background): “Recent work uses audio for an array of video understanding tasks outside of action recognition, including self-supervised representation learning [43, 5, 7, 41, 33, 54], audio-visual source separation [41, 2, 13, 19, 76, 21], localizing sounds in video frames [6, 48, 55], and generating sounds from ”
Co-Separating Sounds of Visual Objects (2019, background): “Localizing Sounds in Video Frames Localization entails identifying the pixels where the sound of a video comes from, but not separating the audio [25, 19, 10, 4, 40, 45].”
LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment (2023, methodology): “For example, we achieved R@1 scores of 10.6 and 10.0 on AVE (Tian et al., 2018) and VGGS, respectively.”
iPerceive: Applying Common-Sense Reasoning to Multi-Modal Dense Video Captioning and Video Question Answering (2020, neutral): “, detection [31], localization [52], tracking [63]).”