To Find Where You Talk: Temporal Sentence Localization in Video with Attention Based Location Regression

By Yitian Yuan, Tao Mei, Wenwu Zhu. AAAI Conference on Artificial Intelligence, 2018

377 citations recorded by the source corpus.

We have witnessed the tremendous growth of videos over the Internet, where most of these videos are typically paired with abundant sentence descriptions, such as video titles, captions and comments. Therefore, it has been increasingly crucial to associate specific video segments with the corresponding informative text descriptions, for a deeper understanding of video content. This motivates us to explore an overlooked problem in the research community — temporal sentence localization in video, which aims to automatically determine the start and end points of a given sentence within a paired video. For solving this problem, we face three critical challenges: (1) preserving the intrinsic temporal structure and global context of video to locate accurate positions over the entire video sequence; (2) fully exploring the sentence semantics to give clear guidance for localization; (3) ensuring the efficiency of the localization method to adapt to long videos. To address these issues, we propose a novel Attention Based Location Regression (ABLR) approach to localize sentence descriptions in videos in an efficient end-to-end manner. Specifically, to preserve the context information, ABLR first encodes both video and sentence via Bi-directional LSTM networks. Then, a multi-modal co-attention mechanism is presented to generate both video and sentence attentions. The former reflects the global video structure, while the latter highlights the sentence details for temporal localization. Finally, a novel attention based location prediction network is designed to regress the temporal coordinates of sentence from the previous attentions. We evaluate the proposed ABLR approach on two public datasets ActivityNet Captions and TACoS. Experimental results show that ABLR significantly outperforms the existing approaches in both effectiveness and efficiency.

Subfields: Computer Vision and Pattern Recognition, Language and Linguistics

View Paper: https://www.semanticscholar.org/paper/31bb920739f22b4865161f75692785decfea470c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Language and Linguistics

Citations in the corpus

Learning Modality Interaction for Temporal Sentence Localization and Event Captioning in Videos (2020, methodology): “For the audio features, we employ the MFCC (Mel-Frequency Cepstral Coeﬃcients) on the captioning task and SoundNet [2] on the sentence localization task.”
Modality Shifting Attention Network for Multi-Modal Video Question Answering (2020, background): “Sundry efforts that have made progress towards binding the two fields include [8, 5, 22, 32] in visual grounding, [31, 30, 36, 25] in image/video captioning, [10, 2, 35, 37] in video moment retrieval, and”
VTimeLLM: Empower LLM to Grasp Video Moments (2023, background): “Traditional approaches can be categorized into two types: proposal-based [4, 29, 32] and proposal-free methods [9, 30, 33].”
Embracing Uncertainty: Decoupling and De-bias for Robust Temporal Grounding (2021, methodology): “First of all, we compare our model DeNet with other state-of-the-art methods using standard metrics on two datasets, which contains CTRL [6], SMRL [38], MAC [8], MLVI [39], CBP [36], SAP [4], MAN [46], 2D-TAN [47], EXCL [9], TMLGA [9], DRN [45], SCDM [43], LGI [24] and ABLR [44].”
Local-Global Video-Text Interactions for Temporal Grounding (2020, methodology): “Second, ABLR relies on coarse interactions between video and text inputs and often fails to capture fine-grained correlations between video segments and query words.”
Span-based Localizing Network for Natural Language Video Localization (2020, methodology): “We adopt “R@ n, IoU = µ ” and “mIoU” as the evaluation metrics, following (Gao et al., 2017; Liu et al., 2018a; Yuan et al., 2019b).”