MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment

By Da Zhang, Xiyang Dai, Xin Eric Wang, Yuan-fang Wang, Larry S. Davis. 2018

344 citations recorded by the source corpus.

This research strives for natural language moment retrieval in long, untrimmed video streams. The problem is not trivial especially when a video contains multiple moments of interests and the language describes complex temporal dependencies, which often happens in real scenarios. We identify two crucial challenges: semantic misalignment and structural misalignment. However, existing approaches treat different moments separately and do not explicitly model complex moment-wise temporal relations. In this paper, we present Moment Alignment Network (MAN), a novel framework that unifies the candidate moment encoding and temporal structural reasoning in a single-shot feed-forward network. MAN naturally assigns candidate moment representations aligned with language semantics over different temporal locations and scales. Most importantly, we propose to explicitly model moment-wise temporal relations as a structured graph and devise an iterative graph adjustment network to jointly learn the best structure in an end-to-end manner. We evaluate the proposed approach on two challenging public benchmarks DiDeMo and Charades-STA, where our MAN significantly outperforms the state-of-the-art by a large margin.

Subfields: Computer Vision and Pattern Recognition, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

Citations in the corpus

Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation (2018, background): “, captioning an image or a video with natural language [9, 10, 44, 46, 52, 53, 47] or localizing desired objects within an image given a natural language description [35, 18, 54, 55].”
Embracing Uncertainty: Decoupling and De-bias for Robust Temporal Grounding (2021, methodology): “First of all, we compare our model DeNet with other state-of-the-art methods using standard metrics on two datasets, which contains CTRL [6], SMRL [38], MAC [8], MLVI [39], CBP [36], SAP [4], MAN [46], 2D-TAN [47], EXCL [9], TMLGA [9], DRN [45], SCDM [43], LGI [24] and ABLR [44].”
COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning (2020, background): “There has been growing interest in temporal localization of natural language in videos [7, 71, 72, 73, 2].”
QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries (2021, methodology): “For moment retrieval, there are work that try to score generated moment proposals [13, 35, 4, 48, 46, 40], predict moment start-end indices [9, 18, 20, 44, 47] or regress moment coordinates [7].”
Span-based Localizing Network for Natural Language Video Localization (2020, methodology): “…task, which is solved by applying multimodal The Ground Truth Moment matching architecture to ﬁnd the best matching video segment for a given language query (Gao et al., 2017; Hendricks et al., 2018; Liu et al., 2018a; Ge et al., 2019; Xu et al., 2019; Chen and Jiang, 2019; Zhang et al., 2019).”
Local-Global Video-Text Interactions for Temporal Grounding (2020, methodology): “Most existing methods [1, 4, 5, 8, 9, 20, 32, 37] tackle the problem typically in the scan-and-localize framework, which in a nutshell compares a query with all candidate proposals of time intervals and selects the one with the highest matching”
Dense Regression Network for Video Grounding (2020, methodology): “[46] explicitly model temporal relations among proposals using a graph.”
VTimeLLM: Empower LLM to Grasp Video Moments (2023, background): “Traditional approaches can be categorized into two types: proposal-based [4, 29, 32] and proposal-free methods [9, 30, 33].”