Video Relationship Reasoning Using Gated Spatio-Temporal Energy Graph

By Yao-Hung Hubert Tsai, Santosh Kumar Divvala, Louis-philippe Morency, Ruslan Salakhutdinov, Ali Farhadi. 2019

112 citations recorded by the source corpus.

Visual relationship reasoning is a crucial yet challenging task for understanding rich interactions across visual concepts. For example, a relationship \{man, open, door\} involves a complex relation \{open\} between concrete entities \{man, door\}. While much of the existing work has studied this problem in the context of still images, understanding visual relationships in videos has received limited attention. Due to their temporal nature, videos enable us to model and reason about a more comprehensive set of visual relationships, such as those requiring multiple (temporal) observations (e.g., \{man, lift up, box\} vs. \{man, put down, box\}), as well as relationships that are often correlated through time (e.g., \{woman, pay, money\} followed by \{woman, buy, coffee\}). In this paper, we construct a Conditional Random Field on a fully-connected spatio-temporal graph that exploits the statistical dependency between relational entities spatially and temporally. We introduce a novel gated energy function parametrization that learns adaptive relations conditioned on visual observations. Our model optimization is computationally efficient, and its space computation complexity is significantly amortized through our proposed parameterization. Experimental results on benchmark video datasets (ImageNet Video and Charades) demonstrate state-of-the-art performance across three standard relationship reasoning tasks: Detection, Tagging, and Recognition.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Continuous Scene Representations for Embodied AI (2022, background): “To overcome this issue, other approaches create scene graphs from videos [13, 28, 33, 36, 45, 49].”
Visual Relation Grounding in Videos (2020, background): “In contrast, relation as a detection task in video domain has earned little attention, partly due to the great challenges in joint video object detection and relation prediction with insufficient video data [26,35].”
Transformer Dissection: An Unified Understanding for Transformer’s Attention via the Lens of Kernel (2019, methodology): “…allows us to explore new family of attention while at the same time offering a framework to categorize previous attention variants (Vaswani et al., 2017; Shaw et al., 2018; Huang et al., 2018b; Dai et al., 2019; Child et al., 2019; Lee et al., 2018; Wang et al., 2018; Tsai et al., 2019a).”
Spatio-Temporal Graph for Video Captioning With Knowledge Distillation (2020, methodology): “On the other hand, modeling object relations via video spatio-temporal graphs [34, 43] has been explored to explicitly construct links between high-level entities by leveraging the relation-modeling nature of graphs.”
Experts Collaboration Learning for Continual Multi-Modal Reasoning (2023, unknown): “[35], [36], [37], [38], [39], [40], [41], [42], [43] and some multi-modal reasoning benchmarks [11], [30], [44] have been developed.”
Beyond Short-Term Snippet: Video Relation Detection With Spatio-Temporal Global Context (2020, methodology): “[36] constructs a similar graph as above, but utilizes Conditional Random Field to exploit the statistical dependency between objects.”