Unified Graph Structured Models for Video Understanding

By Anurag Arnab, Chen Sun, Cordelia Schmid. 2021

62 citations recorded by the source corpus.

Accurate video understanding involves reasoning about the relationships between actors, objects and their environment, often over long temporal intervals. In this paper, we propose a message passing graph neural network that explicitly models these spatio-temporal relations and can use explicit representations of objects, when supervision is available, and implicit representations otherwise. Our formulation generalises previous structured models for video understanding, and allows us to study how different design choices in graph structure and representation affect the model’s performance. We demonstrate our method on two different tasks requiring relational reasoning in videos – spatio-temporal action detection on AVA and UCF101-24, and video scene graph classification on the recent Action Genome dataset – and achieve state-of-the-art results on all three datasets. Furthermore, we show quantitatively and qualitatively how our method is able to more effectively model relationships between relevant entities in the scene.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Graph Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Graph Learning

Citations in the corpus

Multiview Transformers for Video Recognition (2022, methodology): “ViViT [4] extends this to video by extracting N non-overlapping, spatio-temporal “tubelets” [3] from the input video, x1, x2, .”
ViViT: A Video Vision Transformer (2021, background): “Recently, these models were augmented by incorporating self-attention into their later layers to better capture long-range dependencies [75, 23, 79, 1].”