Visual Relation Grounding in Videos

By Junbin Xiao, Xindi Shang, Xun Yang, Sheng Tang, Tat-seng Chua. 2020

45 citations recorded by the source corpus.

In this paper, we explore a novel task named visual Relation Grounding in Videos (vRGV). The task aims at spatio-temporally localizing the given relations in the form of subject-predicate-object in the videos, so as to provide supportive visual facts for other high-level video-language tasks (e.g., video-language grounding and video question answering). The challenges in this task include but not limited to: (1) both the subject and object are required to be spatio-temporally localized to ground a query relation; (2) the temporal dynamic nature of visual relations in videos is difficult to capture; and (3) the grounding should be achieved without any direct supervision in space and time. To ground the relations, we tackle the challenges by collaboratively optimizing two sequences of regions over a constructed hierarchical spatio-temporal region graph through relation attending and reconstruction, in which we further propose a message passing mechanism by spatial attention shifting between visual entities. Experimental results demonstrate that our model can not only outperform baseline approaches significantly, but also produces visually meaningful facts to support visual grounding. (Code is available at this https URL).

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Shuicheng Yan
2. Tat-seng Chua
3. Pan Zhou
4. Junbin Xiao
5. Luc Van Gool
6. Angela Yao
7. Dacheng Tao
8. Xindi Shang

In Multimodal Learning

1. Shuicheng Yan
2. Angela Yao
3. Tat-seng Chua
4. Pan Zhou
5. Xindi Shang
6. Junbin Xiao

Citations in the corpus

• NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, background): “Instead, they are likely better at learning the differences between the provided correct and incorrect answers, which arguably, challenges more on grounding rather than inferring the answers in the videos [49].”
• Video Graph Transformer for Video Question Answering (2022, methodology): “We use different transformations to reflect the asymmetric nature of real-world subject-object interactions [26,58].”