Attention-Based Multimodal Fusion for Video Description

By Chiori Hori, Takaaki Hori, Teng-Yok Lee, Ziming Zhang, Bret A. Harsham, John R. Hershey, Tim K. Marks, Kazuhiro Sumi. 2017

397 citations recorded by the source corpus.

Current methods for video description are based on encoder-decoder sentence generation using recurrent neural networks (RNNs). Recent work has demonstrated the advantages of integrating temporal attention mechanisms into these models, in which the decoder network predicts each word in the description by selectively giving more weight to encoded features from specific time frames. Such methods typically use two different types of features: image features (from an object classification model), and motion features (from an action recognition model), combined by naive concatenation in the model input. Because different feature modalities may carry task-relevant information at different times, fusing them by naive concatenation may limit the model's ability to dynamically determine the relevance of each type of feature to different parts of the description. In this paper, we incorporate audio features in addition to the image and motion features. To fuse these three modalities, we introduce a multimodal attention model that can selectively utilize features from different modalities for each word in the output description. Combining our new multimodal attention model with standard temporal attention outperforms state-of-the-art methods on two standard datasets: YouTube2Text and MSR-VTT.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, neutral): “image/video captioning [130], [242], [243], [244].”
Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering (2019, background): “[9] which proposed to generate video captions by combining different types of features such as video and audio.”
A Case Study on Combining ASR and Visual Features for Generating Instructional Video Captions (2019, background): “In particular, Ramanishka et al. (2016); Xu et al. (2017); Hori et al. (2017); Shen et al. (2017); Chuang et al. (2017); Hao et al. (2018) all report small performance gains when incorporating audio features on top of visual features.”
Progressive Attention Memory Network for Movie Story Question Answering (2019, neutral): “Similar to modality attention [9, 13], dynamic modality fusion is soft-attention based algorithm that determines the contribution of each modality for answering the question.”
Integrating Scene Text and Visual Appearance for Fine-Grained Image Classification (2017, methodology): “The attention mechanism is applied to video description in [76], [77], and [78] studied the spatial attention for image captioning.”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “, 2015), multimodal fusion (Hori et al., 2017), temporal attention (Song et al.”
Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue Systems (2019, background): “Popular research areas include image captioning (Vinyals et al., 2015; Xu et al., 2015), video captioning (Hori et al., 2017; Li et al., 2018) and visual question-answering (QA) (Antol et al., 2015; Goyal et al., 2017).”
Multi-Horizon Time Series Forecasting with Temporal Attention Learning (2019, methodology): “Hori [17] proposed to handle multimodal data by fusing features of different modalities such as texts, audios and videos together with softly assigned weights of each modality.”
A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2021, methodology): “[233] proposed a multimodal attention framework for video captioning and sentence generation based on the encoder–decoder structure using RNNs.”
Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “Nevertheless, the audio modality can serve as complementary information for more reliable and efficient HAR, as shown by the existing works [12], [306].”
Watch, Listen, and Describe: Globally and Locally Aligned Cross-Modal Attentions for Video Captioning (2018, methodology): “Moreover, prior work only employed hand-crafted audio features ( e.g. MFCC) for video cap-tioning (Ramanishka et al., 2016; Xu et al., 2017; Hori et al., 2017).”
Data augmentation techniques for the Video Question Answering task (2020, methodology): “ames are more important based on both the encoded video features and the textual features. In particular, the temporal attention module is based on the works by Bahdanau et al. [2] and by Hori et al. [10]. It receives in input the encoded video features v and the encoded textual features w, and ca”
AudioCaps: Generating Captions for Audios in The Wild (2019, background): “Recently, there have been some efforts to solve video captioning with audio input (Hori et al., 2017, 2018; Wang et al., 2018).”
Attentive Moment Retrieval in Videos (2018, background): “Later, Hori et al.[14] expanded the attention model to selectively attend not just to specific times or spatial regions, but to specific modalities of the inputs such as image features, motion features, and audio features.”
Motion Guided Spatial Attention for Video Captioning (2019, background): “As a result, video captioning approaches (Hori et al. 2017; Xu et al. 2017; Chen et al. 2017) have always used motion information from C3D as just another modality for fusion only.”
Deep Multimodal Feature Encoding for Video Ordering (2020, methodology): “Audio-visual representation learning has seen applications in several tasks such as event classification [40], audiovisual localization [40, 2], biometric matching [35], sound localization [37, 2], person identification [34], action recognition [37], on/off-screen audio separation [12, 37], and vide”
PVNet: A Joint Convolutional Network of Point Cloud and Multi-View for 3D Shape Recognition (2018, methodology): “Inspired by the recent advance of attention mechanism in different tasks [4, 14, 30], we propose a novel fusion method named attention embedding fusion.”
Multimodal SpeakerBeam: Single Channel Target Speech Extraction with Audio-Visual Speaker Clues (2019, background): “Motivated by the success of the attention-based audio-visual fusion in automatic speech recognition [9] and video description [10] tasks, this paper proposes an attention mechanism that integrates the audio and visual speaker clues.”