Motion Guided Spatial Attention for Video Captioning

By Shaoxiang Chen, Yu-Gang Jiang. 2019

133 citations recorded by the source corpus.

Sequence-to-sequence models incorporated with attention mechanism have shown promising improvements on video captioning. While there is rich information both inside and between frames, spatial attention is rarely explored and motion information is usually handled by 3D-CNNs as just another modality for fusion. On the other hand, researches about human perception suggest that apparent motion can attract attention. Motivated by this, we aim to learn spatial attention on video frames under the guidance of motion information for caption generation. We present a novel video captioning framework by utilizing Motion Guided Spatial Attention (MGSA). The proposed MGSA exploits the motion between video frames by learning spatial attention from stacked optical flow images with a custom CNN. To further relate the spatial attention maps of video frames, we designed a Gated Attention Recurrent Unit (GARU) to adaptively incorporate previous attention maps. The whole framework can be trained in an end-to-end manner. We evaluate our approach on two benchmark datasets, MSVD and MSR-VTT. The experiments show that our designed model can generate better video representation and state of the art results are obtained under popular evaluation metrics such as BLEU@4, CIDEr, and METEOR.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Natural Language Generation

Citations in the corpus

End-to-end Generative Pretraining for Multimodal Video Captioning (2022, methodology): “Early works implemented the visual encoder as a 2D CNN (either frozen or finetuned) applied to video frames, which was then naturally extended to 3D CNNs [6,53], to better capture motion dynamics, with temporal aggregation over the entire video typically performed using attention strategies [9].”
Controllable Video Captioning With POS Sequence Guidance Based on Gated Fusion Network (2019, methodology): “In this subsection, we compare our method with the state-of-the-art methods with multiple features on benchmark datasets, including SA [53], M3 [47], v2t navigator [18], Aalto [36], VideoLab [31], MALSTM [51], M&M-TGM [4], PickNet [8], LSTMTSAIV [28], SibNet [23], MGSA [5], and SCNLSTM [14], most of”
Object Relational Graph With Teacher-Recommended Learning for Video Captioning (2020, unknown): “Speciﬁcally, compared with GRU-EVE, MGSA, POS+CG and POS+VCT using the same features as ours, which demonstrate the superior performance without the effects of features.”
Learning Modality Interaction for Temporal Sentence Localization and Event Captioning in Videos (2020, methodology): “Recent works [73,37,9,44,11,43,41] suggest that further improvements can be obtained by additionally leveraging motion and audio representations.”