MAM-RNN: Multi-level Attention Model Based RNN for Video Captioning
By Xuelong Li, Bin Zhao, Xiaoqiang Lu. 2017
99 citations recorded by the source corpus.
Visual information is quite important for the task of video captioning. However, in the video, there are a lot of uncorrelated content, which may cause interference to generate a correct caption. Based on this point, we attempt to exploit the visual features which are most correlated to the caption. In this paper, a Multi-level Attention Model based Recurrent Neural Network (MAM-RNN) is proposed, where MAM is utilized to encode the visual feature and RNN works as the decoder to generate the video caption. During generation, the proposed approach is able to adaptively attend to the salient regions in the frame and the frames correlated to the caption. Practically, the experimental results on two benchmark datasets, i.e., MSVD and Charades, have shown the excellent performance of the proposed approach.
Subfields: Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Dacheng Tao
- Wenhao Jiang
- Wei David Liu
- Shaoxiang Chen
- Chaoyue Wang
- Luc Van Gool
- Yu-Gang Jiang
- Cordelia Schmid
Citations in the corpus
- Towards Interpretable Reinforcement Learning Using Attention Augmented Agents (2019, neutral): “They have been applied with some success to question-answering datasets [7, 8], text translation [5, 9], video classification and captioning [10, 11],”
- Motion Guided Spatial Attention for Video Captioning (2019, methodology): “MAM-RNN (Li, Zhao, and Lu 2017).”
- Syntax-Aware Action Targeting for Video Captioning (2020, methodology): “To increase efficiency as done in [62, 28], we select 28 uniformly-spaced frames from each video clip.”
- Learning Modality Interaction for Temporal Sentence Localization and Event Captioning in Videos (2020, background): “Another group of works [70,33,9] are focused on exploiting spatial information in video frames by applying a dynamic attention mechanism to aggregate frame features spatially.”
- Swin Transformer Embedding UNet for Remote Sensing Image Semantic Segmentation (2022, background): “Zhao et al. [45] and Li et al. [46] presented region-level attention and frame-level attention for video captioning.”