Object-Aware Aggregation With Bidirectional Temporal Graph for Video Captioning

By Junchao Zhang, Yuxin Peng. 2019

191 citations recorded by the source corpus.

Video captioning aims to automatically generate natural language descriptions of video content, which has drawn a lot of attention recent years. Generating accurate and fine-grained captions needs to not only understand the global content of video, but also capture the detailed object information. Meanwhile, video representations have great impact on the quality of generated captions. Thus, it is important for video captioning to capture salient objects with their detailed temporal dynamics, and represent them using discriminative spatio-temporal representations. In this paper, we propose a new video captioning approach based on object-aware aggregation with bidirectional temporal graph (OA-BTG), which captures detailed temporal dynamics for salient objects in video, and learns discriminative spatio-temporal representations by performing object-aware local feature aggregation on detected object regions. The main novelties and advantages are: (1) Bidirectional temporal graph: A bidirectional temporal graph is constructed along and reversely along the temporal order, which provides complementary ways to capture the temporal trajectories for each salient object. (2) Object-aware aggregation: Learnable VLAD (Vector of Locally Aggregated Descriptors) models are constructed on object temporal trajectories and global frame sequence, which performs object-aware aggregation to learn discriminative representations. A hierarchical attention mechanism is also developed to distinguish different contributions of multiple objects. Experiments on two widely-used datasets demonstrate our OA-BTG achieves state-of-the-art performance in terms of BLEU@4, METEOR and CIDEr metrics.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Spatio-Temporal Graph for Video Captioning With Knowledge Distillation (2020, methodology): “[49] propose to perform object tracking and model object trajectories using GRU.”
Object Relational Graph With Teacher-Recommended Learning for Video Captioning (2020, background): “OA-BTG applies a bidirectional temporal graph to capture temporal trajectories for each object.”
Learning Modality Interaction for Temporal Sentence Localization and Event Captioning in Videos (2020, methodology): “Among the competitive methods, OA-BTG [75] utilizes object-level information from an external detector, and MARN [39] uses a more advanced 3D CNN to extract motion features.”
Modality Shifting Attention Network for Multi-Modal Video Question Answering (2020, background): “Sundry efforts that have made progress towards binding the two fields include [8, 5, 22, 32] in visual grounding, [31, 30, 36, 25] in image/video captioning, [10, 2, 35, 37] in video moment retrieval, and”
Open-book Video Captioning with Retrieve-Copy-Generate Network (2021, methodology): “With the advent of the encoder-decoder framework, most of the current work is studying how to better use visual features [35, 23, 1, 36, 22, 24] and design elaborate models [34, 2, 28, 13] to generate sentences directly.”
Syntax-Aware Action Targeting for Video Captioning (2020, methodology): “OA-BTG [65] captures salient objects with their temporal dynamics.”