Learning Multimodal Attention LSTM Networks for Video Captioning

By Jun Xu, Ting Yao, Yongdong Zhang, Tao Mei. 2017

159 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

A Case Study on Combining ASR and Visual Features for Generating Instructional Video Captions (2019, background): “In particular, Ramanishka et al. (2016); Xu et al. (2017); Hori et al. (2017); Shen et al. (2017); Chuang et al. (2017); Hao et al. (2018) all report small performance gains when incorporating audio features on top of visual features.”
Object-Aware Aggregation With Bidirectional Temporal Graph for Video Captioning (2019, methodology): “Following [23, 16], we only consider the English descriptions, taking 1,200, 100, 670 clips for training, validation, testing.”
Syntax-Aware Action Targeting for Video Captioning (2020, methodology): “Speciﬁcally, we choose i) fundamental methods including SA [60], M3 [56], MA-LSTM [59], VideoLab [40], v2t navigator [23], ii) latest state-of-the-art methods including PickNet [6], RecNet local [55], OA-BTG [65], MARN [38], GRU-EVE [1], and POS-CG 3 .”
Attentive Moment Retrieval in Videos (2018, background): “Recently, Xu et al. [42] proposed a multimodal attention LSTM network, which fully exploits both multimodal streams and temporal attention to selectively focus on specific elements during the sentence generation.”
Learning Modality Interaction for Temporal Sentence Localization and Event Captioning in Videos (2020, methodology): “Utilizing multimodal (appearance, motion, and audio) features is also common in recent works, but only a few works [25,76,69,36] tried to handle the relative importances among diﬀer-ent modalities using cross-modal attention.”
Motion Guided Spatial Attention for Video Captioning (2019, methodology): “MA-LSTM (Xu et al. 2017).”
Controllable Video Captioning With POS Sequence Guidance Based on Gated Fusion Network (2019, methodology): “In this subsection, we compare our method with the state-of-the-art methods with multiple features on benchmark datasets, including SA [53], M3 [47], v2t navigator [18], Aalto [36], VideoLab [31], MALSTM [51], M&M-TGM [4], PickNet [8], LSTMTSAIV [28], SibNet [23], MGSA [5], and SCNLSTM [14], most of”
SibNet: Sibling Convolutional Encoder for Video Captioning (2018, background): “It is worth noting that most state-of-the-art methods [5, 9, 25–27, 35, 51, 53, 55] take a combination of multiple complementary features, including frame-level CNN features (ResNet [12], Inception [39], GoogleNet [38]), clip-level CNN features (C3D [40]) and audio features (MFCC [22]), as input to…”
Dense Procedure Captioning in Narrated Instructional Videos (2019, background): “Besides, there are also some works try to incorporate multi-modal information (e.g. audio stream) for dense video captioning task(Ramanishka et al., 2016; Xu et al., 2017; Wang et al., 2018b).”
Watch, Listen, and Describe: Globally and Locally Aligned Cross-Modal Attentions for Video Captioning (2018, methodology): “Moreover, prior work only employed hand-crafted audio features ( e.g. MFCC) for video cap-tioning (Ramanishka et al., 2016; Xu et al., 2017; Hori et al., 2017).”