Spatio-Temporal Dynamics and Semantic Attribute Enriched Visual Encoding for Video Captioning

By Nayyer Aafaq, Naveed Akhtar, Wei David Liu, Syed Zulqarnain Gilani, Ajmal Saeed Mian. 2019

222 citations recorded by the source corpus.

Automatic generation of video captions is a fundamental challenge in computer vision. Recent techniques typically employ a combination of Convolutional Neural Networks (CNNs) and Recursive Neural Networks (RNNs) for video captioning. These methods mainly focus on tailoring sequence learning through RNNs for better caption generation, whereas off-the-shelf visual features are borrowed from CNNs. We argue that careful designing of visual features for this task is equally important, and present a visual feature encoding technique to generate semantically rich captions using Gated Recurrent Units (GRUs). Our method embeds rich temporal dynamics in visual features by hierarchically applying Short Fourier Transform to CNN features of the whole video. It additionally derives high level semantics from an object detector to enrich the representation with spatial dynamics of the detected objects. The final representation is projected to a compact space and fed to a language model. By learning a relatively simple language model comprising two GRU layers, we establish new state-of-the-art on MSVD and MSR-VTT datasets for METEOR and ROUGE_L metrics.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

GIT: A Generative Image-to-text Transformer for Vision and Language (2022, background): “[1] Nayyer Aafaq, Naveed Akhtar, Wei Liu, Syed Zulqarnain Gilani, and Ajmal Mian.”
Recent Advances and Trends in Multimodal Deep Learning: A Review (2021, methodology): “N Aafaq et al. [1] proposed video captioning and capitalized on Spatio-temporal dynamics of videos to extract high-level semantics using 2D and 3D CNNs hierarchically, and GRU is used for the text generation part.”
Object Relational Graph With Teacher-Recommended Learning for Video Captioning (2020, unknown): “9 GRU-EVE [1] 2019 InceptionResnetV2 C3D YOLO 47.”
Learning Modality Interaction for Temporal Sentence Localization and Event Captioning in Videos (2020, methodology): “We adopt the set of features commonly used by recent state-of-the-art methods [59,1,9], which are Inception-ResNet v2 and C3D for visual and motion modalities, respectively.”
Syntax-Aware Action Targeting for Video Captioning (2020, methodology): “…strategy against the current state-of-the-art methods in video captioning that strictly follow the train/val/test splits provided by [52], including the Baseline, S2VT [52], hRNN [63], HRNE [33], LSTM-E [34], SCN-LSTM [15], DMRM [61], LSTM-TSA [35], BAE [3], PickNet [6], M3 [56] and GRU-EVE [1].”
SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning (2021, background): “Recent researches [3, 33, 38, 42, 47] mainly focus on modeling the relationship between fixed video representations and the output textual descriptions via an encoder-decoder framework for video captioning.”
Joint Syntax Representation Learning and Visual Cue Translation for Video Captioning (2019, methodology): “Most existing methods of video captioning [48, 41, 13, 30, 4, 12, 8, 1] mainly focus on investigating various visual perception models by exploiting informative semantics without considering learning of syntax representation for generating sentences.”
Open-book Video Captioning with Retrieve-Copy-Generate Network (2021, methodology): “With the advent of the encoder-decoder framework, most of the current work is studying how to better use visual features [35, 23, 1, 36, 22, 24] and design elaborate models [34, 2, 28, 13] to generate sentences directly.”