Watch, Listen, and Describe: Globally and Locally Aligned Cross-Modal Attentions for Video Captioning

By X. Wang, Yuan-fang Wang, William Yang Wang. 2018

80 citations recorded by the source corpus.

A major challenge for video captioning is to combine audio and visual cues. Existing multi-modal fusion methods have shown encouraging results in video understanding. However, the temporal structures of multiple modalities at different granularities are rarely explored, and how to selectively fuse the multi-modal representations at different levels of details remains uncharted. In this paper, we propose a novel hierarchically aligned cross-modal attention (HACA) framework to learn and selectively fuse both global and local temporal dynamics of different modalities. Furthermore, for the first time, we validate the superior performance of the deep audio features on the video captioning task. Finally, our HACA model significantly outperforms the previous best systems and achieves new state-of-the-art results on the widely used MSR-VTT dataset.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation (2018, background): “Much work [38,31,15,9,40,34,33,32] has been done in language generation conditioned on visual inputs.”
Video Captioning via Hierarchical Reinforcement Learning (2017, background): “Since then, numerous improvements were introduced, such as attention [39, 41], hierarchical recurrent neural network (RNN) [40, 15, 2, 30, 36], C3D features [26], joint embedding space [20], language fusion [5], multi-task learning [17], etc.”
AudioCaps: Generating Captions for Audios in The Wild (2019, background): “MFCC-LSTM does not perform as well as VGGish-LSTM , even with the similar convolutional recurrent encoder.”
Dense Procedure Captioning in Narrated Instructional Videos (2019, background): “Besides, there are also some works try to incorporate multi-modal information (e.g. audio stream) for dense video captioning task(Ramanishka et al., 2016; Xu et al., 2017; Wang et al., 2018b).”
Learning to Compose Topic-Aware Mixture of Experts for Zero-Shot Video Captioning (2018, methodology): “As a result, it has attracted increasing attention in recent years in both NLP (Venugopalan et al. 2016; Wang et al. 2018a) and computer vision communities (Krishna et al. 2017).”