Multilevel Language and Vision Integration for Text-to-Clip Retrieval

By Huijuan Xu, Kun He, Bryan A. Plummer, Leonid Sigal, Stan Sclaroff, Kate Saenko. AAAI Conference on Artificial Intelligence, 2018

369 citations recorded by the source corpus.

We address the problem of text-based activity retrieval in video. Given a sentence describing an activity, our task is to retrieve matching clips from an untrimmed video. To capture the inherent structures present in both text and video, we introduce a multilevel model that integrates vision and language features earlier and more tightly than prior work. First, we inject text features early on when generating clip proposals, to help eliminate unlikely clips and thus speed up processing and boost performance. Second, to learn a fine-grained similarity metric for retrieval, we use visual features to modulate the processing of query sentences at the word level in a recurrent neural network. A multi-task loss is also employed by adding query re-generation as an auxiliary task. Our approach significantly outperforms prior work on two challenging benchmarks: Charades-STA and ActivityNet Captions.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/83b2a55aecd5f917dbedbc0c5ef3ff3b61013958

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval (2020, background): “Existing approaches for SVMR typically involve an expensive early fusion of queries and videos [9, 10, 38, 4, 34].”
Fine-Grained Similarity Measurement between Educational Videos and Exercises (2020, methodology): “Several approaches have been proposed for video segment retrieval and recommendation [24, 39, 44].”
Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models (2021, background): “The topic of compositional learning has been extensively studied in V&L tasks including visual question answering (VQA) [3], image captioning [1, 2] and video retrieval [41].”
Temporal Localization of Moments in Video Collections with Natural Language (2019, methodology): “More recent approaches have integrated alignment of clips with language queries inside a neural network as part of a temporal modular network [23] or joint alignment with temporal attention [4, 5, 24, 35].”
Span-based Localizing Network for Natural Language Video Localization (2020, methodology): “We compare VSLBase and VSLNet with the following state-of-the-arts: CTRL (Gao et al., 2017), ACRN (Liu et al., 2018a), TGN (Chen et al., 2018), ACL-K (Ge et al., 2019), QSPN (Xu et al., 2019), SAP (Chen and Jiang, 2019), MAN (Zhang et al., 2019), SM-RL (Wang et al., 2019), RWM-RL (He et al., 2019), ”
QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries (2021, methodology): “For moment retrieval, there are work that try to score generated moment proposals [13, 35, 4, 48, 46, 40], predict moment start-end indices [9, 18, 20, 44, 47] or regress moment coordinates [7].”
Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language (2019, methodology): “2018c), QSPN (Xu et al. 2019), SM-RL (Wang, Huang, and Wang 2019), SLTA (Jiang et al.”
Local-Global Video-Text Interactions for Temporal Grounding (2020, methodology): “Most existing methods [1, 4, 5, 8, 9, 20, 32, 37] tackle the problem typically in the scan-and-localize framework, which in a nutshell compares a query with all candidate proposals of time intervals and selects the one with the highest matching”
Dense Regression Network for Video Grounding (2020, background): “13 T-to-C [40] C3D 35.”
Learning Modality Interaction for Temporal Sentence Localization and Event Captioning in Videos (2020, methodology): “Since the testing set is not publicly available, we evaluate our method on the validation set as previous works [62,67].”
Embracing Uncertainty: Decoupling and De-bias for Robust Temporal Grounding (2021, methodology): “First of all, we compare our model DeNet with other state-of-the-art methods using standard metrics on two datasets, which contains CTRL [6], SMRL [38], MAC [8], MLVI [39], CBP [36], SAP [4], MAN [46], 2D-TAN [47], EXCL [9], TMLGA [9], DRN [45], SCDM [43], LGI [24] and ABLR [44].”
MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment (2018, methodology): “Until recently, several methods were proposed to closely integrate language and video representation [54, 6]: Xu et al.”