Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language

By Songyang Zhang, Houwen Peng, Jianlong Fu, Jiebo Luo. 2019

606 citations recorded by the source corpus.

We address the problem of retrieving a specific moment from an untrimmed video by a query sentence. This is a challenging problem because a target moment may take place in relations to other temporal moments in the untrimmed video. Existing methods cannot tackle this challenge well since they consider temporal moments individually and neglect the temporal dependencies. In this paper, we model the temporal relations between video moments by a two-dimensional map, where one dimension indicates the starting time of a moment and the other indicates the end time. This 2D temporal map can cover diverse video moments with different lengths, while representing their adjacent relations. Based on the 2D map, we propose a Temporal Adjacent Network (2D-TAN), a single-shot framework for moment localization. It is capable of encoding the adjacent temporal relation, while learning discriminative features for matching video moments with referring expressions. We evaluate the proposed 2D-TAN on three challenging benchmarks, i.e., Charades-STA, ActivityNet Captions, and TACoS, where our 2D-TAN outperforms the state-of-the-art.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “[236] Songyang Zhang, Houwen Peng, Jianlong Fu, and Jiebo Luo.”
QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries (2021, methodology): “For moment retrieval, there are work that try to score generated moment proposals [13, 35, 4, 48, 46, 40], predict moment start-end indices [9, 18, 20, 44, 47] or regress moment coordinates [7].”
Embracing Uncertainty: Decoupling and De-bias for Robust Temporal Grounding (2021, methodology): “First of all, we compare our model DeNet with other state-of-the-art methods using standard metrics on two datasets, which contains CTRL [6], SMRL [38], MAC [8], MLVI [39], CBP [36], SAP [4], MAN [46], 2D-TAN [47], EXCL [9], TMLGA [9], DRN [45], SCDM [43], LGI [24] and ABLR [44].”
Modality Shifting Attention Network for Multi-Modal Video Question Answering (2020, background): “Sundry efforts that have made progress towards binding the two fields include [8, 5, 22, 32] in visual grounding, [31, 30, 36, 25] in image/video captioning, [10, 2, 35, 37] in video moment retrieval, and”
Egocentric Video-Language Pretraining (2022, methodology): “Vis-text PT R@ 1 R@ 5 R@ 1 R@ 5 2D-TAN [44] The grey highlighted rows correspond to zero-shot evaluation . to the egocentric data availability and the video-text interaction learned from large-scale pretraining.”