Span-based Localizing Network for Natural Language Video Localization

By Hao Zhang, Aixin Sun, Wei Jing, Joey Tianyi Zhou. 2020

402 citations recorded by the source corpus.

Given an untrimmed video and a text query, natural language video localization (NLVL) is to locate a matching span from the video that semantically corresponds to the query. Existing solutions formulate NLVL either as a ranking task and apply multimodal matching architecture, or as a regression task to directly regress the target video span. In this work, we address NLVL task with a span-based QA approach by treating the input video as text passage. We propose a video span localizing network (VSLNet), on top of the standard span-based QA framework, to address NLVL. The proposed VSLNet tackles the differences between NLVL and span-based QA through a simple and yet effective query-guided highlighting (QGH) strategy. The QGH guides VSLNet to search for matching video span within a highlighted region. Through extensive experiments on three benchmark datasets, we show that the proposed VSLNet outperforms the state-of-the-art methods; and adopting span-based QA framework is a promising direction to solve NLVL.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Mohit Bansal
3. Jitendra Malik
4. Andrew Zisserman
5. Dacheng Tao
6. Trevor Darrell
7. Cordelia Schmid
8. Ming-Hsuan Yang

In Multimodal Learning

1. Jie Lei
2. Mohit Bansal
3. Kevin Qinghong Lin
4. Tamara L. Berg
5. Alex Wang
6. Mike Zheng Shou
7. Rui Yan
8. Zhe Gan

Citations in the corpus

• Sequence Parallelism: Long Sequence Training from System Perspective (2021, background): “Recently, Transformer also achieved promising results on computer vision tasks [3, 25, 26] and even on bioinformatics tasks [4, 18].”
• QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries (2021, background): “For moment retrieval, there are work that try to score generated moment proposals [13, 35, 4, 48, 46, 40], predict moment start-end indices [9, 18, 20, 44, 47] or regress moment coordinates [7].”
• VTimeLLM: Empower LLM to Grasp Video Moments (2023, background): “Traditional approaches can be categorized into two types: proposal-based [4, 29, 32] and proposal-free methods [9, 30, 33].”
• Egocentric Video-Language Pretraining (2022, methodology): “62 VSLNet [45] Frozen CC3M+WebVid-2M 5.”