Temporally Grounding Natural Sentence in Video

By Jingyuan Chen, Xinpeng Chen, Lin Ma, Zequn Jie, Tat-seng Chua. Conference on Empirical Methods in Natural Language Processing, 2018

342 citations recorded by the source corpus.

We introduce an effective and efficient method that grounds (i.e., localizes) natural sentences in long, untrimmed video sequences. Specifically, a novel Temporal GroundNet (TGN) is proposed to temporally capture the evolving fine-grained frame-by-word interactions between video and sentence. TGN sequentially scores a set of temporal candidates ended at each frame based on the exploited frame-by-word interactions, and finally grounds the segment corresponding to the sentence. Unlike traditional methods treating the overlapping segments separately in a sliding window fashion, TGN aggregates the historical information and generates the final grounding result in one single pass. We extensively evaluate our proposed TGN on three public datasets with significant improvements over the state-of-the-arts. We further show the consistent effectiveness and efficiency of TGN through an ablation study and a runtime test.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/452aca244ef62a533d8b46a54c6212fe9fa3ce9a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Temporal Localization of Moments in Video Collections with Natural Language (2019, methodology): “In recent work [4, 5, 10, 23, 24, 35], a joint model over language and video features is used to return the alignment cost,”
Large-Scale Representation Learning from Visually Grounded Untranscribed Speech (2019, methodology): “Unlike Chrupała et al. (2017), who similarly applied TTS to MS-COCO (Chen et al.”
Span-based Localizing Network for Natural Language Video Localization (2020, methodology): “We compare VSLBase and VSLNet with the following state-of-the-arts: CTRL (Gao et al., 2017), ACRN (Liu et al., 2018a), TGN (Chen et al., 2018), ACL-K (Ge et al., 2019), QSPN (Xu et al., 2019), SAP (Chen and Jiang, 2019), MAN (Zhang et al., 2019), SM-RL (Wang et al., 2019), RWM-RL (He et al., 2019), ”
MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment (2018, methodology): “methods that use temporal modular network [29], finegrained frame-by-word attentions [6] and temporal contextual encoding [18].”
Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language (2019, methodology): “, who design a recurrent module to temporally capture the evolving finegrained frame-by-word interactions between video and sentence (Chen et al. 2018).”
TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval (2020, background): “Existing approaches for SVMR typically involve an expensive early fusion of queries and videos [9, 10, 38, 4, 34].”
Learning Modality Interaction for Temporal Sentence Localization and Event Captioning in Videos (2020, background): “However, modality interactions are largely overlooked in the research of high-level video understanding tasks, such as event captioning [71,57,58] and temporal sentence localization [17,34,8].”
Violin: A Large-Scale Dataset for Video-and-Language Inference (2020, neutral): “Video-and-Language Research With the emergence of large-scale video datasets [6, 1, 29, 11, 58], several videoand-language tasks have been proposed, such as video captioning [21, 56, 62, 18, 33, 16, 47, 59], localizing video segments from natural language queries [19, 3, 8, 37], video reasoning [65]”
Dense Regression Network for Video Grounding (2020, methodology): “[4] propose to select the starting and ending frames by leveraging cross-modal interactions between text and video.”
VTimeLLM: Empower LLM to Grasp Video Moments (2023, background): “Traditional approaches can be categorized into two types: proposal-based [4, 29, 32] and proposal-free methods [9, 30, 33].”
Local-Global Video-Text Interactions for Temporal Grounding (2020, methodology): “Most existing methods [1, 4, 5, 8, 9, 20, 32, 37] tackle the problem typically in the scan-and-localize framework, which in a nutshell compares a query with all candidate proposals of time intervals and selects the one with the highest matching”