Attentive Moment Retrieval in Videos

By Meng Liu, Xiang Wang, Liqiang Nie, Xiangnan He, Baoquan Chen, Tat-seng Chua. Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 2018

340 citations recorded by the source corpus.

In the past few years, language-based video retrieval has attracted a lot of attention. However, as a natural extension, localizing the specific video moments within a video given a description query is seldom explored. Although these two tasks look similar, the latter is more challenging due to two main reasons: 1) The former task only needs to judge whether the query occurs in a video and returns an entire video, but the latter is expected to judge which moment within a video matches the query and accurately returns the start and end points of the moment. Due to the fact that different moments in a video have varying durations and diverse spatial-temporal characteristics, uncovering the underlying moments is highly challenging. 2) As for the key component of relevance estimation, the former usually embeds a video and the query into a common space to compute the relevance score. However, the later task concerns moment localization where not only the features of a specific moment matter, but the context information of the moment also contributes a lot. For example, the query may contain temporal constraint words, such as "first'', therefore need temporal context to properly comprehend them. To address these issues, we develop an Attentive Cross-Modal Retrieval Network. In particular, we design a memory attention mechanism to emphasize the visual features mentioned in the query and simultaneously incorporate their context. In the light of this, we obtain the augmented moment representation. Meanwhile, a cross-modal fusion sub-network learns both the intra-modality and inter-modality dynamics, which can enhance the learning of moment-query representation. We evaluate our method on two datasets: DiDeMo and TACoS. Extensive experiments show the effectiveness of our model as compared to the state-of-the-art methods.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/519da94369c1d87e09c592f239b55cc9486b5b7c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

To Find Where You Talk: Temporal Sentence Localization in Video with Attention Based Location Regression (2018, methodology): “2017) or extending the clip boundaries with a predefined scale (Gao et al. 2017; Liu et al. 2018).”
Embracing Uncertainty: Decoupling and De-bias for Robust Temporal Grounding (2021, methodology): “[22] adopt a sliding window to combine each centralclip feature and its context-clip features as one proposal candidate.”
Dynamic Modality Interaction Modeling for Image-Text Retrieval (2021, background): “It benefits a variety of applications, ranging from cross-modal retrieval [11, 12, 23, 41] tomultimedia recommendation [36].”
Self-supervised pre-training and contrastive representation learning for multiple-choice video QA (2020, background): “2018), video moment retrieval (Liu et al. 2018), and video question answering (video QA) (Lei et al.”
Span-based Localizing Network for Natural Language Video Localization (2020, methodology): “…task, which is solved by applying multimodal The Ground Truth Moment matching architecture to ﬁnd the best matching video segment for a given language query (Gao et al., 2017; Hendricks et al., 2018; Liu et al., 2018a; Ge et al., 2019; Xu et al., 2019; Chen and Jiang, 2019; Zhang et al., 2019).”
Dense Regression Network for Video Grounding (2020, neutral): “00 ACRN [29] 16.”
Learning Modality Interaction for Temporal Sentence Localization and Event Captioning in Videos (2020, background): “However, modality interactions are largely overlooked in the research of high-level video understanding tasks, such as event captioning [71,57,58] and temporal sentence localization [17,34,8].”
Temporal Localization of Moments in Video Collections with Natural Language (2019, methodology): “In recent work [4, 5, 10, 23, 24, 35], a joint model over language and video features is used to return the alignment cost,”
Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language (2019, methodology): “Most of the current language-queried moment localization models follow a two-step pipeline (Gao et al. 2017; Hendricks et al. 2017; Ge et al. 2019; Liu et al. 2018b; Song and Han 2018).”