Local-Global Video-Text Interactions for Temporal Grounding
By Jonghwan Mun, Minsu Cho, Bohyung Han. Computer Vision and Pattern Recognition, 2020
354 citations recorded by the source corpus.
This paper addresses the problem of text-to-video temporal grounding, which aims to identify the time interval in a video semantically relevant to a text query. We tackle this problem using a novel regression-based model that learns to extract a collection of mid-level features for semantic phrases in a text query, which corresponds to important semantic entities described in the query (e.g., actors, objects, and actions), and reflect bi-modal interactions between the linguistic features of the query and the visual features of the video in multiple levels. The proposed method effectively predicts the target time interval by exploiting contextual information from local to global during bi-modal interactions. Through in-depth ablation studies, we find out that incorporating both local and global context in video and text interactions is crucial to the accurate grounding. Our experiment shows that the proposed method outperforms the state of the arts on Charades-STA and ActivityNet Captions datasets by large margins, 7.44\% and 4.61\% points at Recall@tIoU=0.5 metric, respectively.
Subfields: Computer Vision and Pattern Recognition
View Paper: https://www.semanticscholar.org/paper/7e3d5b20e5df692deb80d9e100e4f34c1a8f8031
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Cordelia Schmid
- Luc Van Gool
- Andrew Zisserman
- Ming-Hsuan Yang
- Andrea Vedaldi
- Jitendra Malik
- Kristen Grauman
- Xiaogang Wang
Citations in the corpus
- Embracing Uncertainty: Decoupling and De-bias for Robust Temporal Grounding (2021, methodology): “Recently, some approaches [24, 45, 28, 11] adopt a proposal-free framework.”
- COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning (2020, background): “There has been growing interest in temporal localization of natural language in videos [7, 71, 72, 73, 2].”
- Look Before you Speak: Visually Contextualized Utterances (2020, background): “Popular vision and language tasks include visual question answering [4, 5, 22, 28, 58, 61], visual dialog [15, 16, 36, 40, 67], visual captioning [56, 59, 68, 73, 86], visual grounding [18, 22, 57, 83] and video-text retrieval [7, 25, 49, 53, 63].”